Volume Number: 23 (2007)
Issue Number: 06
Column Tag: MacEnterprise

Troubleshooting Directory Services

The basics

By Philip Rinehart, Yale University

One of the most common questions that are discussed on the MacEnterpise mailing list is the diagnosis and analysis of integration of OS X with Directory Services. While most commonly talked about in reference to Active Directory, many of the tools that can be used are applicable to any Directory Service, as most rely on the same core foundation. The first step of any analysis and troubleshooting is based on having a solid Domain Naming System, commonly known as DNS.

DNS problems

How does one go about troubleshooting DNS on OS X? More importantly, what should be tested and in what order? Checking forward and reverse DNS records usually makes the most sense initially. Misconfigured DNS information can often be the root of Directory Service problems. If the administrator configuring DNS has not correctly set both the forward and reverse DNS records, any attempt to bind or use a Directory Service becomes incrementally difficult.

Testing forward DNS

The first step is to test forward DNS records. As a brief reminder, forward DNS is the process of looking up a domain name and having the IP address returned. There are many tools to test this functionality, but one that can quickly test forward DNS resolution is the command line tool, host. Open a terminal, and type in host name.domain.com. If the forward DNS record is properly set up, the return should be: host name.domain.com has address 111.222.333.444. Easy, right?

Testing reverse DNS

Next, test reverse DNS records. Reverse DNS is the process of taking an IP address, and resolving it to a domain name. As before, using the host command line tool is easiest. Open a terminal, and type in host 111.222.333.444. If the reverse DNS record is properly set up, the return should be: host 444.333.222.111.in-addr.arpa domain name pointer name.domain.com. It cannot be emphasized enough how important it is to have correctly configured forward and reverse DNS records. In particular, the Active Directory plug-in can be very sensitive to incorrect DNS configuration. Generally, if DNS is functioning correctly, binding to any Directory Service should be trivial. So, if DNS is working, what should be the next step?

At times, testing DNS using host can point to the cause of Directory Service problems. However, it is very important to note that testing this is only testing servers providing the DNS records, not necessarily the resolution by the client machine.

The first, and most obvious way to test network connectivity is with the use of ping. Ping the server providing directory services, and determine whether the client has connectivity with the provider. Silly as it sounds, check the cable or IP address being provided by a wireless server. Often the simple things are the solution!

Next, using the tool lookupd, client side DNS resolution can be tested. Testing reverse DNS, using the interactive debug mode, can verify that the results returned by using host are what the client is using as well. Invoke the debugger with lookupd -d. Following the same testing procedure, first enter:

hostWithName: hostname.domain.com

Next, enter:

hostWithInternetAddress: 111.222.333.444

Both commands will return a lot of information, including the agent that is being used by the client, as well as cache information, and how many hits have occurred by the operating system. This information can point to subtle DNS problems if it is different from the information returned by the host command.

If a machine is successfully bound, the next most common problem that is reported is the inability to authenticate to a directory service exhibited by a shaking login window. Let's tackle the simplest way to test authentication first.

Dirt

Dirt? Never heard of the tool? It is a lesser-known tool that can be used to test Directory Services from the command line. It is particularly useful when used to test authentication against a bound Directory Service. The syntax can be a little tricky, but when used, it can be quite powerful. The first test is to check that the user exists in the Directory Services node. Here's how, open a terminal and type:

dirt -u username -n

The return value should be:

User username was found in:

/DSNode

This test simply does a quick verification of the username that is being used to login, and makes sure that the OS X client can see the information contained by the Directory Node. Next, test user authentication using the node name as follows:

dirt -m "/Active Directory/All Domains" ¬
-u activedirectoryusername -p activedirectorypassword

The command above specifically tests Active Directory, but any valid domain node can be tested. Some important notes:

The -u flag uses the username from the node you are testing against, in the above example it is the Active Directory username

The -p flag uses the password from the username that is being tested. In the above example, it is the Active Directory account password

The node is the Search node as referenced in Directory Access. In the above example, all Active Directory domains are searched.

Local administrative privileges are not required to use this tool.

Currently, the password must be entered with the -p option, as omitting it does not work as documented in the manual page. The return value can provide very useful troubleshooting information. As an example, this output is returned with a bad password:

Call to dsGetRecordList returned count = 1 with Status : eDSNoErr : (0)
Username: username
Password: password
Error : eDSAuthFailed : (-14090)

This return status very clearly reflects the failed password authentication. Let's dive even deeper.

Kerberos is increasingly being used for authentication for many Directory Services. If the password is correct, but the shaking login window is still occurring, the next area to focus on, especially for Active Directory and Open Directory, is Kerberos troubleshooting. Fortunately, testing is painless using the command tool, kinit. Type in the following:

kinit username

On failure, a very understandable error message is displayed:

Kerberos Login Failed: Clock skew too big. 
Please check your time, time zone and daylight savings settings.

From this error message, it is clear that Kerberos is failing because the clock differential, referred to as skew, is too great. While many administrators set the clock to use a network timeserver, it is not uncommon for OS X to drift by more than five minutes; this is usually greater than the allowable clock skew. Adjust the date and time to be within five minutes of the Directory Services authentication source, and this problem is solved!

These problems are the 'low-hanging' fruit, and can usually quickly solve Directory Service problems. What about problems that are more difficult?

What if none of these troubleshooting steps works? There are three ways to log information, one for Directory Services, one for Managed Client (MCX), and one for Portable Home Directories.

Directory Services

If none of the quick steps provides an answer, debugging Directory Services is often needed to troubleshoot particularly complex problems. There are two different levels of logging which can be invoked on demand USR1, and USR2. Both are turned on similarly, with the command:

killall -USR1 DirectoryService

USR1 will log information to

/Library/Logs/DirectoryService/DirectoryService.debug.log.

USR2 sends all information to the system.log file. One last thing, both levels can also be set by touching a file in the following directory as follows:

touch /Library/Preferences/DirectoryService/.DSLogDebugAtStart (USR1)
touch /Library/Preferences/DirectoryService/.DSLogAPIAtStart (USR2)

Reboot the client, and debugging will begin at boot time.

Managed Client Services (MCX)

Debugging information can be collected and set using a command line only option:

defaults write /Library/Preferences/com.apple.MCXDebug debugOutput 3

Writing this preference will log all information relating to client management to the system.log file. Three is the maximum value that can be set. Using this value, a tremendous amount of information can be seen in the client log, and may point to managed client problems.

Portable Home Directories

Portable home directories can also be logged with a hidden preference:

defaults write com.apple.MirrorAgent debugOutput 4

This preference will log all portable home directory information to ~/Library/Logs/MirrorAgent.log. Again, though not fun, this information can often point to the source of trouble when attempting to diagnose a sticky Portable Home Directory problem.

Diagnosing and troubleshooting can be one of the most complex issues in the integration of OS X clients into heterogeneous network infrastructures. Remember to always start from the simplest explanation, as it often is the source of the problem. It is easy to errantly assume a much larger problem, when, in fact, the problem may be quite simple. However, with the above techniques and tools, bending Directory Services to your will should be far simpler.

Philip Rinehart is co-chair of the steering committee leading the Mac OS X Enterprise Project (macenterprise.org) and is the Lead Mac Analyst at Yale University. He has been using Macintosh Computers since the days of the Macintosh SE, and Mac OS X since its Developer Preview Release. Before coming to Yale, he worked as a Unix system administrator for a dot-com company. He can be reached at: philip.rinehart@yale.edu.

The MacEnterprise project is a community of IT professionals sharing information and solutions to support Macs in an enterprise. We collaborate on the deployment, management, and integration of Mac OS X client and server computers into multi-platform computing environments