TweetFollow Us on Twitter

OS X Investigation and Troubleshooting

Volume Number: 22 (2006)
Issue Number: 5
Column Tag: Programming

Mac In The Shell

OS X Investigation and Troubleshooting

by Edward Marczak

The Secrets to OS X success

"How did you know that?" A question I'm often asked. Usually right after pulling out some arcane bit of OS X knowledge. Now, I hardly know everything - far, far from it. But, I try to stay a little ahead of the curve. What you're reading now is part 1 of a multi-part column on learning the depths of OS X. Of course, the deeper you dig, the more quickly you can troubleshoot the system. At the end, I hope you will have picked up some new tips and tricks. Of course, we need to begin at the beginning.

The Basics

One of the first things I like to install on my own machines is Tripwire (this, by the way, goes for end-user stations and servers). Long known as a security tool, a tripwire will take a snapshot of file system, and then report any changes made to that system. From a security perspective, that's incredibly important:, especially when you see something change in an area that shouldn't ever change! It is also a great way to learn about your Mac. What changes every day? Did the patch to your software install exactly what it claims? (And that's it...I'm not going to launch into the history of the product this time!)

Currently, I run the "Tripwire" tripwire (there are others out there). You can download an OS X binary from <http://www.macguru.net/~frodo/Tripwire-osx.html>. Go forth, download and install. Once you've installed it, the tricky part is the configuration and setup, and that's what I'll cover here. But, I will breeze through the install.

Please note that the binaries in the download are PPC only. If you're on an Intel Mac, grab the source, and compile it up yourself. Who knows, there may be a pre-packaged version somewhere by the time this column runs.

This is a full command-line install, so fire up Terminal.app (or iTerm, or...). Uncompress the tarball, get root (sudo bash, sudo -s, su -...take your pick), and run ./install.sh. Press ENTER to read through the license agreement (space, space, space, space, space, space), and agree. Note that:

This program will copy Tripwire files to the following directories:
           
           TWBIN:  /usr/local/tripwire/bin
           TWMAN:  /usr/local/tripwire/man
        TWPOLICY:  /usr/local/tripwire/policy
        TWREPORT:  /usr/local/tripwire/report 
            TWDB:  /usr/local/tripwire/db
    TWSITEKEYDIR:  /usr/local/tripwire/key
   TWLOCALKEYDIR:  /usr/local/tripwire/key

Follow the rest of the instructions, and definitely initialize the database when asked, even though it will undoubtedly take a while. Tripwire is a deep, and somewhat complex product. I've been using it longer than I can remember on both servers, where I install it with a little more consciousness to security, and my personal workstations. Tripwire usage alone, could take an article or two. A quick Google search turned up this good tutorial - http://www.weberdev.com/Manuals/rhl-rg-en-80/ch-tripwire.html - and I recommend you read it, if you want to get into tripwire deeper than I present here. Just remember to adjust paths in your head for the install you just did.

Tripwire operates against a file policy. You should notice that all files were installed under /usr/local. In /usr/local/tripwire/policy, you'll find two files, tw.pol and twpol.txt. The first, is a signed binary file - the one tripwire uses to run from. The second, is just a text file. To change the policy, you need to sign a text file into a binary using your passphrase. Use twpol.txt as a guide. For the most part, the default policy is just dated, and you can comment out anything relating to System 9. Make your changes to your policy, change to the /usr/local/tripwire/bin directory and run:

# ./twadmin -m P ../policy/twpol.txt
Please enter your site passphrase: 
Wrote policy file: /usr/local/tripwire/policy/tw.pol

Since this is security software, we can't just allow any change to policy, right? We need to re-initialize the baseline snapshot. So, if you've chosen to do this, run ./tripwire -m i from the /usr/local/tripwire/bin directory and enter your passphrase. Now you're using your custom policy. From there, when you run tripwire -m c, a report will be output to your terminal and to /usr/local/tripwire/report, where you can pick up a text-based report.

Tripwire just gives me that extra happy feeling that I know what's going on with my machine. When applications piggyback on another's install (yes Smart Crash Reports, I'm looking in your direction...), it won't surprise you later on. I would like to share how I automate Tripwire reports, since it may not be entirely obvious. Here's a portion of the shell script that I had perform some nightly maintenance:

## Check and report on differences
/usr/local/sbin/tripwire -m c > /var/root/logs/`date +%Y%m%d`.txt
## Update the database
echo "***********************" >> /var/root/logs/`date +%Y%m%d`.txt
/usr/local/sbin/tripwire -m u -a -r /usr/local/lib/tripwire/report/`ls 
/usr/local/lib/tripwire/report` -P "My Passphrase Here" -v >> 
/var/root/logs/`date +%Y%m%d`.txt

First thing that gets done is a report, which is redirected to a file - the name of which is based on the date. Then, after writing a marker to my log, I update the Tripwire database with a new snapshot, so I'm ready for the next night. (You might notice from this snippet that I don't have things in the exact same place you might. So if you want to steal this, make sure you get the paths right!)

While I still do rely on Tripwire for high-level changes, please make note that it's far from perfect in a Macintosh environment - especially with Tiger. Tripwire is slightly aged at this point, and worked well when it arrived on the scene. However, it will miss changes in HFS+ metadata, like keywords for Spotlight and ACL information. Just understand that Tripwire is no longer an ideal security solution for the Mac (if it ever was).

Find Out

The find command - something I've been meaning to dig into, in a column somewhere. Sounds simple, right? find finds files. However, there's an impressive array of options that let you narrow down the scope of your results. Of course, you can find by name:

find . -name "report" -print

You have to tell find where to begin looking, that's the first "." - start in the current directory. From there, you have to give find its criteria. In this case, we're looking for a name. Finally, we have to tell it what to do with the items it finds. Here, we just want it printed to our terminal. That's OK, but not for the purposes of this article. You should hit the man page for all of the options that find contains, but we'll look at some practical OS X example usage. Here's one of the more useful ones: find files updated since boot. Since OS X creates the /mach file at every boot, we have a great marker to use as a time stamp. If you want to find all files in the /System hierarchy that have been updated since boot, use this:

# find /System -newer /mach -print

On my machine, this currently yields this:

/System/Library/Caches/com.apple.kernelcaches
/System/Library/CoreServices/RemoteManagement/ARDAgent.app/Contents/Support
/System/Library/CoreServices/RemoteManagement/ARDAgent.app/Contents/Support/networksetup
/System/Library/CoreServices/RemoteManagement/ARDAgent.app/Contents/Support/systemsetup

The kernelcaches file comes up because, well, I was fiddling with kernel extensions. The ARD files got modified because I needed to fire up ARD to get back into my machine. (looooong story why I had to configure and fire up ARD through ssh on my own machine...)

If you want to find files that have been modified since a certain time, not necessarily boot, use touch to drop a marker, and use find against that. Something like this:

# touch -t 200601011300.00 marker
# ls -l marker 
-rw-r--r--   1 root  wheel  0 Jan  1 13:00 marker
# find / -newer marker -print

This will find all files created or modified on my machine since the first of January, 2006, 1pm. Since I suspect that would be a fairly high number, I'll skip the output and leave that as an exercise for the reader.

Of course, with no constraints, find will just return everything under a certain hierarchy, which comes in handy for quick before and after snapshots. find /Library -print > ~/liblist.txt will print out all files in /Library, and redirect the output to a file in your home directory named "liblist.txt". Run that before installing software, and again, with a different capture file name, after the software is installed, and compare the two. You'll find any new files that the installer may have dropped into /Library.

That's not an exhaustive look at find, as it's not the sole focus of this article. But, make no mistake - find is incredibly useful. If you've ever examined the locate.updatedb script that runs weekly, you'll see that it builds its database, using find. The deeper you dig, the more uses you'll find.

What's da BOM?

Speaking of file tracking and installation, did you know that the Apple installer will happily show you the files it will install before it installs them? Really. Next time you need to install software, look for "Show Files" under the File menu (or press Apple-I). That's part of the package's bill of materials. Figure 1 shows the beginning of the bill of materials for Viva Designer.



Figure 1: Installer showing a bill of materials

In conjunction with our other techniques above, this is a handy way to see where an installer may want to spray files. Additionally, if you end up running Tripwire, the file changes it reports, should match up with the BOM that an installer presents. If not, someone is lying!

You can also determine the bill of materials from the command-line, if you are installing or inspecting a package remotely. The lsbom binary will display the contents of a BOM archive. Witness:

$ cd VivaDesigner-Free-5.1.0-4055.pkg/Contents
$ ls -la
total 149696
drwxr-xr-x   7    marczak   marczak        238    Feb 15 11:13 .
drwxr-xr-x   3    marczak   marczak        102    Feb 15 11:13 ..
-r--r--r--      1    marczak   marczak      46540    Feb 15 11:13 Archive.bom
-r--r--r--      1    marczak   marczak   76584362    Feb 15 11:13 Archive.pax.gz
-r--r--r--      1    marczak   marczak       1373    Feb 15 11:13 Info.plist
-r--r--r--      1    marczak   marczak          8    Feb 15 11:13 PkgInfo
drwxr-xr-x   12   marczak   marczak        408    Feb 17 15:34 Resources
$ lsbom Archive.bom
.       40755   501/80
./Applications  40755   501/80
./Applications/Viva     40777   0/80
./Applications/Viva/AddIns      40777   0/80
./Applications/Viva/AddIns/RTF Import   40777   0/80
./Applications/Viva/AddIns/RTF Import/Resources 40777   0/80
./Applications/Viva/AddIns/RTF Import/Resources/ansi-gen        100666  0/80    3231    2569670117
./Applications/Viva/AddIns/RTF Import/Resources/ansi-sym        100666  0/80    1498    2978768719
./Applications/Viva/AddIns/RTF Import/Resources/ansi.code       100666  0/80    1449    784701331
./Applications/Viva/AddIns/RTF Import/Resources/mac-gen 100666  0/80    3137    1114243711
...
(output clipped for sanity)

If you've never looked at the contents of a package, take a look again at the previous listing. Archive.bom is the packages bill of materials. Archive.pax.gz contains the files themselves! So, if you ever need to grab one file from a package, that's where you can get it from.

Processes

The next-to-last last thing I'm going to delve into this month is the process model of OS X - an important area of understanding for the advanced topics later on. Despite Apple pushing the notion that OS X is Unix, it's not quite, really. It's a mach kernel with Unix-like behavior and APIs. This makes all of that Unix source code compile neatly (mostly), but you're still always operating under the monolithic mach kernel, which does things a little differently than the traditional *BSD, Sys V, and derivative Unix-like works such as Linux and IRIX kernels. All in all, it's a unique mix of a known kernel, a modified BSD Unix that rides on top, and unique parts from Apple that haven't been seen before.

Describing the Mach and microkernel architecture would take a book by itself (one that I would guess exists already), but its foundations are important to understand if we're to troubleshoot deeply. I'm going to run us through the talking points, and the highlights that get us to OS X.

Mach came about after Unix was already in existence. From that perspective, it could see the good points of Unix and use them, and the downsides of Unix and avoid them. Mach was originally developed at Carnegie Mellon University, leapfrogging off of a BSD Unix core. Little by little, Mach replaced parts of the BSD core. To keep compatibility, much of BSD remained in the Mach kernel. Mach v3 moved all BSD code outside of the kernel, resulting in the microkernel featured today. The goal of a microkernel architecture allows the kernel to provide a minimal amount of services, and extensions that run up in userland. Interestingly, this provides a system that allows other operating systems to sit on top. That's one of Mach's primary goals: a simple, extensible kernel. Note, finally, that traditionally, when talking about the kernel, you'd only be referring to the microkernel itself. Apple, with Mac OS X, gets a little more liberal with the definition, as they've forged their own path. When Apple refers to the kernel, that primarily encompasses the Mach kernel, BSD, and I/O Kit. This is done for valid performance reasons.

Here's the important part: Mach's execution environment is called a task. Other Unicies (like the System V, traditionally) break an executing program down into a process. A process allows the kernel to keep track of:

  • * Context - the current location of program execution

  • * The program's credentials (rights)

  • * Memory space that the program has allocated/access to

...but that's not exactly what we're interested in. Mach abstracts things a little differently. A task provides the address space for execution. There is no such thing as a "process" in Mach! A thread is the basic unit of execution. A thread runs inside a task. A task does nothing unless it has a thread running inside it. A task allows communication with the rest of the system via ports (these have nothing to do with IP ports!). Threads communicate over ports via messages.

A task with just one thread running, is similar to a Unix process. The fork system call creates a new process under Unix, and it creates a new task under Mach. So, a task provides virtual memory space, and ports for the threads that are running inside of it. Tasks and threads can be in only one of two states: running and suspended. Operating on a task affects all threads in the task. Mach allows for kernel tasks and threads, and of course, userland tasks and threads.

There you have it: a ridiculously simplified view of Mach. The important point to take away: A task is either running in the kernel or in userland. Thanks to the BSD roots, and Apple's bundling of BSD in kernel-space, you're still going to see plenty of references to "processes" - don't be confused by that - the BSD in Apple's kernel space still references processes. These are retrofitted into OS X by associating a process to a Mach task.

Listing All Open Files

The last utility I'm going to cover this month wraps up everything we've talked about: lsof (list open files). First, one must remember that Unix treats just about everything as a file. So this command, if you're not already familiar with it, may do more than you expect. Go on, get a shell and try it. Just type lsof by itself. You got an absolute ton of output, right? Things that certainly don't look like files, for sure. If you run this command as root, you get everything - everyone else gets a little less. (This is a compile-time option that, thankfully, Apple chose to enable). Specifically, as non-root, you'll only see processes that you have credentials to see. Of course, when troubleshooting, grep comes in extra-handy here (and there are plenty of switches that modify lsof's output). Let's look at a snippet of the files that Word has open, while I type this month's column:



(That's only a handful of the 195 files that actually were listed!) What is all of that? Let's look at a shorter listing:

$ lsof | head -7
COMMAND     PID   USER   FD    TYPE    DEVICE   SIZE/OFF   NODE     NAME
kernel_ta   0     root   cwd   VDIR    14,2     1360       2        /
launchd     1     root   cwd   VDIR    14,2     1360       2        /
launchd     1     root   txt   VREG    14,2     80112      2471328  /sbin/launchd
launchd     1     root   txt   VREG    14,2     1165460    4986063  /usr/lib/dyld
launchd     1     root   txt   VREG    14,2     4314524    7229742  /usr/lib/libSystem.B.dylib
launchd     1     root   0r    VCHR     3,2     0t0        47460484 /dev/null

The command column lists the name of the process that holds a file open. Well, at least the process' first 9 characters by default. That can be changed with a switch. Next is the PID, or, process ID column. The user column lists the user ID, or the ID number of the user that owns the respective process. The remainder of the columns may require a little deeper explanation.

FD is the file descriptor number of the file or one of the following:

  • cwd current working directory

  • jld jail directory

  • ltx shared library text (code and data)

  • Mxx hex memory-mapped type number xx

  • mem memory-mapped file

  • mmap memory-mapped device

  • pd parent directory

  • rtd root directory

  • txt program text (code and data)

The file descriptor number may be followed by a character (see the final line in the example listing above), which has the meaning:

    r - file is open for read.

    w - file is open for write.

    u - file is open for read/write.

    space (no character) - unknown mode, no lock character.

    - (hyphen) - unknown mode and lock character follows.

    The lock character will be one of the following:

    N for an NFS lock of unknown type;

    r for read lock on part of the file;

    R for a read lock on the entire file;

    w for a write lock on part of the file;

    W for a write lock on the entire file;

    u for a read and write lock of any length;

    U for a lock of unknown type;

The type column lists what type of file is open. While lsof can report on many different types, it makes the most sense to concentrate on the types you'll see most:

    FIFO - A FIFO pipe. Much like a regular pipe, but operates as part of the file system and can be accessed by multiple processes. man 1 mkfifo, if you need to know.

    IPv4 - An open IPv4 socket.

    IPv6 - An open IPv6 file.

    KQUEUE - A kernel event queue file. man 2 kevent if you're really interested.

    PIPE - An open unix pipe.

    PSXSEM - Posix semaphore file. A semaphore is like a lock, but with a little more control. With a semaphore, more than one thread can be performing a given operation at once, whereas a lock will restrict operations to a single thread.

    PSXSHM - Posix shared memory.

    VCHR - a character device.

    VDIR - a directory on the filesystem.

    VREG - a regular file on the filesystem.

    VGER - that thing from Star Trek (oh, wait...you won't see that in lsof).

    LINK - a symbolic link.

    systm - a system domain socket.

    unix - a unix domain socket.

The device column is an important one: it tells you which device said file is open on. On OS X, possibly not a big deal as you may be running with a single disk (as I am on my PowerBook at the moment). However, OS X Server may present you with more possibilities (as I hope you're separating the system and user data on a server...but that's another article). The listed 'device' may look a little odd. In some cases, it will be a memory address (in the case of PSXSHM, for example). Files and directories will list the device node number. A device node number looks something like this: 14, 2 - it is listed in the size column. Perhaps this pleads for further explanation.

Device files live in the /dev directory. Take a peek in there and you'll see files that are pretty much like none other on the system. In the permissions column, where you'd expect to either find a 'd' denoting a directory, or a '-' denoting a file, we see instead a 'c' or 'b'. Those represent character or block devices. A character, or raw, device is something like a tty (teletype terminal - what you're using when you fire up Terminal or ssh into another machine). A block device is typically used for a disk or tape device (OK, I still wish OS X had raw tape support...). Without getting too deep into this, a block device gets a buffer assigned by the kernel, and allows you to perform non-sequential access. A character device typically gets used where you'd be reading a stream of information (like from a serial port). What about those crazy numbers?

The numbers in the size column are called the devices major and minor numbers. All of these device entries represent device drivers. The actual driver is either compiled into the kernel (/mach_kernel) or loaded as an extension. The /dev entry is just a pointer to the driver in kernel space. Just because there is an entry in /dev, does not mean that there's a corresponding driver in the kernel. The major number represents the kind of device, while the minor number represents the specific part of that device we're interested in. Let's take a look at some examples:

brw-r-----   1 root   operator   14,   0 Mar  7 16:40 disk0
br--r-----    1 root   operator   14,   1 Mar  7 16:40 disk0s1
brw-r-----   1 root   operator   14,   2 Mar  7 16:40 disk0s3

From this listing, we can immediately see that disk0, disk0s1, and disk0s3 are all block devices with major number 14. Notice what differentiates each of the devices: the minor number - representing a different slice on the disk. Let's look at another snippet:

crw--w----   1 root     tty   4,   0 Mar  7 16:42 ttyp0
crw-------   1 marczak  tty   4,   1 Mar 14 06:26 ttyp1
crw--w----   1 marczak  tty   4,   2 Mar  7 16:46 ttyp2
crw--w----   1 marczak  tty   4,   3 Mar 13 19:20 ttyp3

Not only can we immediately see that these are character devices, but, as expected, have a different major number. Once again, the differentiating factor for each of the ttyp entries is the minor number, each addressing a different ttyp.

Take away this: entries in /dev are not device drivers, nor are they code, but rather, they are simple pointers. Creating an entry in /dev does not create code in the kernel to support the device. You create these special entries with mknod, but you should never have to touch a thing in /dev. Of course, how does this fit into our discussion about lsof? You'll notice that many times in a listing, the device entry will be a major and minor combination. Now you know what that means! Just go look it up in /dev if things aren't adding up. In my case, I have many files open on 14,2. So, I'd do this:

# ls -l /dev | grep "14, *2"
brw-r-----  1 root  operator   14,   2 Mar  7 16:40 disk0s3
crw-r-----  1 root  operator   14,   2 Mar  7 16:40 rdisk0s3

Ah, of course! That makes sense: 14,2 represents my main (and only, at the moment) disk.

Following the device column is the "Size/Off" column. This column shows the size of the file opened, if it is an actual file, the offset into the file, depending on the file type - look for the 0t or 0x prefix - or, possibly no value if lsof can't make a determination, or, is not appropriate.

The next-to-last column is "Node". This will list the file's node number on a local disk, the inode of an NFS file, the Internet protocol type, or possibly nothing, depending on the file type.

We made it! Whew! Last column: Name. This is yet another column whose contents will change based on what type of file is being displayed, and there are many possibilities. I'm going to cover ones that you're most likely to see. For a regular file or directory, the full path name to the file or directory will be displayed. In other cases, we may be looking at a block or character device. Network connections will be listed with appropriate information.

Now, lsof is an incredible utility. Do understand that we're lucky that Apple includes it with OS X: lsof is not included with most Unix distributions. It's not a "built-in," rather; it's an add-on. As such, it works its magic by digging in where it can, making inferences and generally peeking where most utilities don't. It works slightly differently on different varieties of Unix. What I'm getting at here is that it may not be 1000% accurate. Don't let that fill you with doubt, though, as it does a better job than anything out there, really. But there may be a minority of situations where it just doesn't pull up a file, or grabs data from the kernel cache that no longer reflects reality. For the purposes of files that a given process is accessing, though, I haven't had any issues to complain about.

Temporary Stop

We covered a lot of ground this month. All of this is to get you a little more intimate, intertwined, and aware of the system that is OS X. Many times, troubleshooting and learning about OS X comes down to figuring out where a file is or what files have changed within a certain period of time. Next month, we'll carry on using this column as a foundation.

Media of the month: Go rent (or take off your shelf - you know who you are) Tron. Seriously. Incredibly far ahead of its time. If you watch it after having read this column, it should connect a few more synapses.

Until next month, dig in, experiment and enjoy!

References

Mac OS X Tech O: Core OS

http://developer.apple.com/documentation/MacOSX/Conceptual/OSX_Technology_Overview...

Kernel Programming Guide:

http://developer.apple.com/documentation/Darwin/Conceptual/KernelProgramming/index.html


Ed Marczak owns and operates Radiotope, a consultancy that assists companies with technology planning, and implementation. He helps guide business leaders around the pitfalls of technology, and to find ways of connecting them with their clients. Guidance at http://www.radiotope.com.

 

Community Search:
MacTech Search:

Software Updates via MacUpdate

GarageSale 7.0.7 - Create outstanding eB...
GarageSale is a slick, full-featured client application for the eBay online auction system. Create and manage your auctions with ease. With GarageSale, you can create, edit, track, and manage... Read more
SpamSieve 2.9.28 - Robust spam filter fo...
SpamSieve is a robust spam filter for major email clients that uses powerful Bayesian spam filtering. SpamSieve understands what your spam looks like in order to block it all, but also learns what... Read more
Thunderbird 45.7.1 - Email client from M...
As of July 2012, Thunderbird has transitioned to a new governance model, with new features being developed by the broader free software and open source community, and security fixes and improvements... Read more
Opera 43.0.2442.991 - High-performance W...
Opera is a fast and secure browser trusted by millions of users. With the intuitive interface, Speed Dial and visual bookmarks for organizing favorite sites, news feature with fresh, relevant content... Read more
OnyX 3.2.4 - Maintenance and optimizatio...
OnyX is a multifunction utility that you can use to verify the startup disk and the structure of its system files, to run miscellaneous maintenance and cleaning tasks, to configure parameters in the... Read more
VueScan 9.5.71 - Scanner software with a...
VueScan is a scanning program that works with most high-quality flatbed and film scanners to produce scans that have excellent color fidelity and color balance. VueScan is easy to use, and has... Read more
Slack 2.5.1 - Collaborative communicatio...
Slack is a collaborative communication app that simplifies real-time messaging, archiving, and search for modern working teams. Version 2.5.1: New The way we load teams you don't view often has been... Read more
HandBrake 1.0.3 - Versatile video encode...
HandBrake is a tool for converting video from nearly any format to a selection of modern, widely supported codecs. Features Supported Sources VIDEO_TS folder, DVD image or real DVD (unencrypted... Read more
Vivaldi 1.7.735.46 - An advanced browser...
Vivaldi is a browser for our friends. In 1994, two programmers started working on a web browser. Our idea was to make a really fast browser, capable of running on limited hardware, keeping in mind... Read more
Vivaldi 1.7.735.46 - An advanced browser...
Vivaldi is a browser for our friends. In 1994, two programmers started working on a web browser. Our idea was to make a really fast browser, capable of running on limited hardware, keeping in mind... Read more

Mudd Masher arrives this week
Atooi Games, the minds behind Totes the Goat and Mutant Mudds, have a new game in the works -- Mudd Masher. The game, a hybrid of the independent studio's first two titles, is expected to launch this week on March 2. [Read more] | Read more »
The best sales on the App Store this wee...
The App Store has quite an exciting lineup of discount games this week that range across a variety of genres. It's a great opportunity to catch up on some of the premium games you may have been holding off on -- and some you can even grab for free... | Read more »
The best new games we played this week
Ah, here we are again at the close of another busy week. Don't rest too easy, though. We had a lot of great new releases in mobile games this week, and now you're going to have to spend all weekend playing them. That shouldn't be too much of a... | Read more »
Rollercoaster Tycoon Touch Guide: How to...
| Read more »
Rabbids Crazy Rush Guide: How to unlock...
The Rabbids are back in a new endless running adventure, Rabbids Crazy Rush. It's more ridiculous cartoon craziness as you help the little furballs gather enough fuel (soda) to get to the moon. Sure, it's a silly idea, but everyone has dreams --... | Read more »
Tavern Guardians (Games)
Tavern Guardians 1.0 Device: iOS Universal Category: Games Price: $2.99, Version: 1.0 (iTunes) Description: Tavern Guardians is a Hack-and-Slash action game played in the style of a match-three. You can experience high pace action... | Read more »
Slay your way to glory in idle RPG Endle...
It’s a golden age for idle games on the mobile market, and those addictive little clickers have a new best friend. South Korean developer Ekkorr released Endless Frontier last year, and players have been idling away the hours in the company of its... | Read more »
Tiny Striker: World Football Guide - How...
| Read more »
Good news everyone! Futurama: Worlds of...
Futurama is finding a new home on mobile in TinyCo and Fox Interactive's new game, Futurama: Worlds of Tomorrow. They're really doing it up, bringing on board Futurama creator Matt Groening along with the original cast and writers. TinyCo wants... | Read more »
MUL.MASH.TAB.BA.GAL.GAL (Games)
MUL.MASH.TAB.BA.GAL.GAL 1.0 Device: iOS Universal Category: Games Price: $2.99, Version: 1.0 (iTunes) Description: ENDLESS UPGRADES. CONSTANT DANGER. ANCIENT WISDOM. BOUNCY BALLS. Launch Sale, 40% OFF for a very limited time!!! MUL.... | Read more »

Price Scanner via MacPrices.net

27-inch 3.3GHz 5K iMac on sale for $2099, sav...
B&H Photo has the 27″ 3.3GHz 5K Apple iMac on sale for $2099.99 including free shipping plus NY sales tax only. Their price is $200 off MSRP. Amazon also has the 27″ 3.3GHz 5K iMac on sale for $... Read more
21-inch iMacs on sale for up to $111 off MSRP
B&H Photo has select 21″ Apple iMacs on sale for up to $110 off MSRP, each including free shipping plus NY sales tax only: - 21″ 2.8GHz iMac: $1189 $110 off MSRP - 21″ 1.6GHz iMac: $999 $100 off... Read more
12-inch 1.2GHz Retina MacBooks on sale for $2...
Newegg has the 12″ 1.2GHz Space Gray Retina MacBook (sku MLH82LL/A) on sale for $1349.99 including free shipping. Their price is $250 off MSRP, and it’s the lowest price available for this model.... Read more
13-inch MacBook Airs on sale for $100 off MSR...
B&H Photo has 13″ MacBook Airs on sale for $100 off MSRP. Shipping is free, and B&H charges NY sales tax only: - 13″ 1.6GHz/128GB MacBook Air (MMGF2LL/A): $899 $100 off MSRP - 13″ 1.6GHz/... Read more
9-inch 32GB Silver iPad Pro on sale for $549,...
B&H Photo has the 9.7″ 32GB Silver Apple iPad Pro on sale for $549 for a limited time. Shipping is free, and B&H charges NY sales tax only. Their price is $50 off standard MSRP for this model... Read more
13-inch 2.0GHz Apple MacBook Pros on sale for...
B&H has the non-Touch Bar 13″ 2.0GHz MacBook Pros in stock today and on sale for $100 off MSRP. Shipping is free, and B&H charges NY sales tax only: - 13″ 2.0GHz MacBook Pro Space Gray (... Read more
15-inch Touch Bar MacBook Pros on sale for up...
B&H Photo has the new 2016 15″ Apple Touch Bar MacBook Pros in stock today and on sale for up to $150 off MSRP. Shipping is free, and B&H charges NY sales tax only: - 15″ 2.7GHz Touch Bar... Read more
12-inch Retina MacBooks on sale for $1150, $1...
B&H has 12″ 1.1GHz Retina MacBooks on sale for $150 off MSRP. Shipping is free, and B&H charges NY sales tax only: - 12″ 1.1GHz Space Gray Retina MacBook: $1149 $150 off MSRP - 12″ 1.1GHz... Read more
Apple restocks refurbished 11-inch MacBook Ai...
Apple has Certified Refurbished 11″ MacBook Airs (the latest models recently discontinued by Apple), available for up to $170 off original MSRP. An Apple one-year warranty is included with each... Read more
Apple Park Opens to Employees in April With T...
Apple has announced that Apple Park, the company’s new 175-acre campus, will be ready for employees to begin occupying in April. The process of moving more than 12,000 people will take over six... Read more

Jobs Board

*Apple* Solutions Consultant - Apple (United...
# Apple Solutions Consultant Job Number: 55676865 Los Angeles, California, United States Posted: Feb. 22, 2017 Weekly Hours: 40.00 **Job Summary** As an Apple Read more
Programmer/Editor *Apple* Music Dance - App...
# Programmer/Editor Apple Music Dance Job Number: 55565967 Culver City, California, United States Posted: Feb. 23, 2017 Weekly Hours: **Job Summary** Apple Music Read more
Digital Marketing Specialist - *Apple* iClo...
# Digital Marketing Specialist - Apple iCloud Job Number: 54729233 Culver City, California, United States Posted: Feb. 22, 2017 Weekly Hours: 40.00 **Job Summary** Read more
Marketing Specialist, iTunes & *Apple*...
# Marketing Specialist, iTunes & Apple Music Job Number: 55704205 Culver City, California, United States Posted: Feb. 23, 2017 Weekly Hours: 40.00 **Job Summary** Read more
*Apple* Wireless Lead - T-ROC - The Retail O...
…of knowledge in wireless sales and activations to the Beautiful and NEW APPLE Experiencestore within MACYS. THIS role, APPLE Wireless Lead, isbrandnewas MACYS Read more
All contents are Copyright 1984-2011 by Xplain Corporation. All rights reserved. Theme designed by Icreon.