NTP Debugging Techniques

Once the NTP software distribution has been compiled and installed and the configuration file constructed, the next step is to verify correct operation and fix any bugs that may result. Usually, the command line that starts the daemon is included in the system startup file, so it is executed only at system boot time; however, the daemon can be stopped and restarted from root at any time. Usually, no command-line arguments are required, unless special actions described in the ntpd page are required. Once started, the daemon will begin sending messages, as specified in the configuration file, and interpreting received messages.

The best way to verify correct operation is using the ntpq and ntpdc utility programs, either on the server itself or from another machine elsewhere in the network. The ntpq program implements the management functions specified in Appendix A of the NTP specification RFC-1305, Appendix A. The ntpdc program implements additional functions not provided in the standard. Both programs can be used to inspect the state variables defined in the specification and, in the case of ntpdc, additional ones of interest. In addition, the ntpdc program can be used to selectively enable and disable some functions of the daemon while the daemon is running.

In extreme cases with elusive bugs, the daemon can operate in two modes, depending on the presence of the -d command-line debug switch. If not present, the daemon detaches from the controlling terminal and proceeds autonomously. If one or more -d switches are present, the daemon does not detach and generates special output useful for debugging. In general, interpretation of this output requires reference to the sources. However, a single -d does produce only mildly cryptic output and can be very useful in finding problems with configuration and network troubles. With a little experience, the volume of output can be reduced by piping the output to grep and specifying a keyword, such as receive or clock of the trace you want to see.

Some problems are immediately apparent when the daemon first starts running. The most common of these are the lack of ntp (UDP port 123) in the host /etc/services file. Note that NTP always uses UDP and never uses TCP. Other problems are apparent in the system log file. The log file should show the startup banner, some cryptic initialization data, and the computed precision value. The next most common problem is incorrect DNS names. Check that each DNS name used in the configuration file responds to the Unix ping command.

When first started, the daemon normally polls the servers listed in the configuration file at 64-second intervals. In order to allow a sufficient number of samples for the NTP algorithms to reliably discriminate between correctly operating servers (truechimers) and possible intruders (falsetickers), at least four valid messages from at least one server is required before the daemon can set the local clock. However, if the current local time is more than 1000 seconds in error from the server time, the daemon will not set the local clock; instead, it will plant a message in the system log and shut down. It is necessary to set the local clock to within 1000 seconds first, either from the time-of-year (TOY) chip, the ntpdate program or manually by eyeball and wristwatch.

After starting the daemon, run the ntpq program using the -n switch, which will avoid possible distractions due to name resolution problems. Use the pe command to display a billboard showing the status of configured peers and possibly other clients poking the daemon. After operating for a few minutes, the display should be something like:

ntpq>pe
remote      refid       st t when poll reach   delay  offset   disp
===================================================================
+128.4.2.6  132.249.16.1 2 u  131  256   373    9.89   16.28  23.25
*128.4.1.20 .WWVB.       1 u  137  256   377  280.62   21.74  20.23
-128.8.2.88 128.8.10.1   2 u  49   128   376  294.14    5.94  17.47
+128.4.2.17 .WWVB.       1 u  173  256   377  279.95   20.56  16.40

The tattletale character at the left margin displays the synchronization status of each peer. The currently selected peer is marked *, while additional peers designated acceptable for synchronization, but not currently selected, are marked +. Peers marked * and + are included in a weighted average computation to set the local clock; the data produced by peers marked with other symbols are discarded. See the ntpq documentation for the meaning of these symbols.

Additional details for each peer separately can be determined by the following procedure. First, use the as command to display an index of association identifiers, such as

ntpq>as
ind assID status conf reach auth condition last_event cnt
=========================================================
 1  11670   7414   no   yes   ok candidate  reachable   1
 2  11673   7614   no   yes   ok sys.peer   reachable   1
 3  11833   7314   no   yes   ok outlyer    reachable   1
 4  11868   7414   no   yes   ok candidate  reachable   1

ntpq>rv 11670
status=7414 reach, auth, sel_sync, 1 event, event_reach
srcadr=128.4.2.6, srcport=123, dstadr=128.4.2.7, dstport=123, keyid=1,
stratum=2, precision=-10, rootdelay=362.00, rootdispersion=21.99,
refid=132.249.16.1,
reftime=af00bb44.849b0000 Fri, Jan 15 1993 4:25:40.517,
delay= 9.89, offset= 16.28,
dispersion=23.25, reach=373, valid=8,
hmode=2, pmode=1, hpoll=8, ppoll=10, leap=00, flash=0x0,
org=af00bb48.31a90000 Fri, Jan 15 1993 4:25:44.193,
rec=af00bb48.305e3000 Fri, Jan 15 1993 4:25:44.188,
xmt=af00bb1e.16689000 Fri, Jan 15 1993 4:25:02.087,
filtdelay=  16.40 9.89  140.08  9.63  9.72  9.22 10.79 122.99,
filtoffset= 13.24 16.28 -49.19 16.04 16.83 16.49 16.95 -39.43,
filterror=  16.27 20.17  27.98 31.89 35.80 39.70 43.61  47.52

Finally, the state of the local clock can be determined using the rv command (without the argument), such as

ntpq>rv
status=0664 leap_none, sync_ntp, 6 events, event_peer/strat_chg
system="UNIX", leap=00, stratum=2, rootdelay=280.62,
rootdispersion=45.26, peer=11673, refid=128.4.1.20,
reftime=af00bb42.56111000 Fri, Jan 15 1993 4:25:38.336,
poll=8, clock=af00bbcd.8a5de000 Fri, Jan 15 1993 4:27:57.540,
phase=21.147, freq=13319.46, compliance=2

When nothing seems to happen in the pe billboard after some minutes, there may be a network problem. The most common network problem is an access controlled router on the path to the selected peer. Few public NTP time server selectively restricts access, although this may change in future; however, many private network servers do. It also may be the case that the server is down or running in unsynchronized mode due to a local problem. Use the ntpq program and hosts command program to spy on the server variables in the same way you can spy on your own.

Once the daemon has set the local clock, it will continuously track the discrepancy between local time and NTP time and adjust the local clock accordingly. There are two components of this adjustment, time and frequency. These adjustments are automatically determined by the clock discipline algorithm. The behavior of this algorithm is carefully controlled to minimize residual errors due to network jitter and frequency variations of the local clock hardware oscillator due to temperature changes. However, when started for the first time, the algorithm may take many hours to converge to the intrinsic frequency of the host machine.

It has sometimes been the experience that the local clock oscillator frequency error is too large for the NTP discipline algorithm, which cannot correct frequency errors larger than 500 parts-per-million (PPM) or 43 seconds per day. There are two possibilities that may result in this problem. First, the hardware TOY clock chip must be disabled when using NTP, since it can destabilize the discipline process. This is usually done using the tickadj program and the -s command line argument, but other means may be necessary. For instance, in the Sun Solaris kernel, this can be done using a command in the /etc/system system startup file.

Normally, the NTP daemon amortizes time adjustments at a rate of 500 PPM until the remaining adjustment is completely amortized. The adjustment process is updated with a new time correction each second. This process, or slew, operates continuously as long as the adjustment does not exceed 128 milliseconds, which for most Internet paths is a rare event. If an adjustment does exceed this value, perhaps due to an occasional network delay spike, the correction is simply discarded; however, if outlyer values persist in subsequent updates for an interval of 900 seconds, the local clock is stepped to the new value. This behavior is designed to resist errors due to severely congested network paths, as well as errors due to confused radio clocks upon the epoch of a leap second.

Network Problems

Most problems observed with NTP over the years has been some kind of network problem. Frequently, the problem turns out to be a filter in the firewall between a public server and a client on a private network. In other cases the problem is due simply to the fact a designated server is unreachable or itself unsynchronized. The Unix ping program can be used to verify server reachability and the traceroute utility to explore the path between the remote server and local client. The techniques used are common to other network services and considered beyond the scope of the discussion here.

Sometimes such things as access control and authentication can get in the way of successful synchronization. The -d command line debug flag is a useful diagnostic in such cases. One or more debug flags and the Unix grep filter can be used to identify and resolve intractable cases. For this purpose, the lavish commentary sprinkled in the ntp_proto.c source file may be helpful in resolving the cause. In particular, the various header and data sanity checks can be determined for each arriving message.

The Unix tcpdump program is useful to discover rogue packets and to calibrate the traffic between a client and its server or peer. The ntptrace program is useful to discover the synchronization path to the primary server, which can be useful should loops develop. If kernel support is available, the ntptime program can be used to display and alter the kernel state variables. The status bits reveal the current state; see the /usr/include/sys/timex.h header file for interpretation. Typical performance expectations are jitter less than 5 microseconds and stability less than .05 PPM.

With Extreme Prejudice

It has sometimes been the experience that the local clock oscillator frequency error is too large for the NTP discipline algorithm, which can correct frequency errors as large as 500 parts-per-million (PPM) or 43 seconds per day. There are two possibilities that may result in this behavior. First, the hardware time-of-year (TOY) clock chip must be disabled when using NTP, since this can destabilize the discipline process. If this is not done, the time can be jerked from under NTP, with result excessive jitter and instability. The TOY chip can be disabled in many systems using the tickadj program and the -s command line argument, but other means may be necessary. For instance, in the Sun Solaris kernel, this is done using a command in the /etc/system system startup file.

Second, the local intrinsic clock frequency error may exceed the 500 PPM limit. If this occurs, the apparent clock offset will grow until reaching 128 ms; then, after the 900-second timeout, the clock will be stepped to the apparent time. If the -x option is used to always slew the clock, the offset will grow continuously. Eventually, the offset will reach 1000 seconds and the daemon will fall on its sword and terminate with an uncivil message to the system log.

The tickadj program can be used to force the clock frequency error to a low value which the NTP daemon can handle. This is done by changing the value of the tick kernel variable which defines the number of microseconds added to the clock at each timer interrupt. The best way to determine if this is the cause is to start the daemon with a configuration file listing a known good server and including the line disable ntp. For this purpose, the drift file /etc/ntp.drift should be temporarily removed or renamed. If kernel support is available and enabled, previous debugging may have left nonsense in the kernel variables. The ntptime program can be used to set the frequency variable to zero, or the system can be rebooted with the same effect.

Once the daemon has started, the ntpq program can be used to inspect the clock filter values, which show the roundtrip delay, offset and dispersion. By inspecting successive offsets in the eight-sample clock filter window and finding the interval between updates, the intrinsic clock frequency can be calculated. If this exceeds 500 PPM, the daemon cannot correct the frequency and a correction to the tick kernel variable is required. For 100-Hz kernels, which represent the majority of systems, this value is normally 10000, which results in ten milliseconds clock increment for each timer interrupt. Changing this value in unit increments results in frequency changes in 100 PPM increments. The intrinsic offset calculated previously, the correct value of tick can be determined to the nearest 100-PPM value.

Debugging Checklist

1. Verify the /etc/services file host machine is configured to accept UDP packets on the NTP port 123. NTP is specifically designed to use UDP and does not respond to TCP.
2. Check the system log for ntpd messages about configuration errors, name-lookup failures or initialization problems.
3. Using the ntpdc program and iostats command, verify that the received packets and packets sent counters are incrementing. If the packets sent counter does not increment and the configuration file includes designated servers, something may be wrong in the network configuration of the ntpd host. If this counter does increment and packets are actually being sent to the network, but the received packets counter does not increment, something may be wrong in the network or the server may not be responding.
4. If both the packets sent counter and received packets counter do increment, but the rec timestamp in the pe billboard shows far from the current date, received packets are probably being discarded for some reason. There is a handy, undocumented state variable flash visible in the pe billboard. The value is in hex and normally has the value zero (OK). However, if something is wrong, the bits of this variable, reading from the right, correspond to the sanity checks listed in Section 3.4.3 of the NTP specification RFC-1305. A bit other than zero indicates the associated sanity check failed.
5. If the org, rec and xmt timestamps in the pe billboard appear current, but the local clock is not set, as indicated by a stratum number less than 16 in the rv command without arguments, verify that valid clock offset, roundtrip delay and dispersion are displayed for at least one peer. The clock offset should be less than 1000 seconds, the roundtrip delay less than one second and the dispersion less than one second.
6. While the algorithm can tolerate a relatively large frequency error (up to 500 parts per million or 43 seconds per day), various configuration errors (and in some cases kernel bugs) can exceed this tolerance, leading to erratic behavior. This can result in frequent loss of synchronization, together with wildly swinging offsets. Use the ntpdc program (or temporary configuration file) and disable pll command to prevent the ntpd daemon from setting the clock. Using the ntpq or ntpdc programs, watch the apparent offset as it varies over time to determine the intrinsic frequency error. If the error increases by more than 22 milliseconds per 64-second poll interval, the intrinsic frequency must be reduced by some means. The easiest way to do this is with the tickadj program and the -t command line argument.