Description
This article describes a guideline and commands to troubleshoot any NTP synchronization issue on FortiGate and FortiSwitch devices
Scope
FortiGate, FortiSwitch.
Solution
If the FortiGate is not able to sync the time with the configured NTP server, use the following commands to check the NTP server status:
get sys stat
execute date
execute time
diagnose sys ntp status
An example output of the NTP status command is seen below:
diagnose sys ntp status
synchronized: yes, ntpsync: enabled, server-mode: disabled
ipv4 server(time.google.com) 216.239.35.0 -- reachable(0xff) S:1 T:5
server-version=4, stratum=1
reference time is e18b5929.fc81eb59 -- UTC Fri Nov 29 08:45:29 2019
clock offset is 0.011700 sec, root delay is 0.000000 sec
root dispersion is 0.000183 sec, peer dispersion is 5 msec
ipv4 server(time.google.com) 216.239.35.4 -- reachable(0xff) S:1 T:5 selected
server-version=4, stratum=1
reference time is e18b5929.c0f08d85 -- UTC Fri Nov 29 08:45:29 2019
clock offset is 0.009796 sec, root delay is 0.000000 sec
root dispersion is 0.000214 sec, peer dispersion is 8 msec
ipv4 server(time.google.com) 216.239.35.12 -- reachable(0xff) S:1 T:5
server-version=4, stratum=1
reference time is e18b592a.21f2218 -- UTC Fri Nov 29 08:45:30 2019
clock offset is 0.009603 sec, root delay is 0.000000 sec
root dispersion is 0.000214 sec, peer dispersion is 5 msec
ipv6 server(time.google.com) 2001:4860:4806:8:: -- reachable(0xfe) S:0 T:4
no data
ipv6 server(time.google.com) 2001:4860:4806:4:: -- reachable(0xfe) S:0 T:4
no data
ipv6 server(time.google.com) 2001:4860:4806:: -- reachable(0xfe) S:0 T:4
no data
ipv6 server(time.google.com) 2001:4860:4806:c:: -- reachable(0xfe) S:0 T:4
no data
ipv4 server(time.google.com) 216.239.35.8 -- reachable(0xff) S:1 T:5
server-version=4, stratum=1
reference time is e18b592a.bcbcf -- UTC Fri Nov 29 08:45:30 2019
clock offset is 0.013359 sec, root delay is 0.000000 sec
root dispersion is 0.000198 sec, peer dispersion is 3 msec
execute time
current time is: 16:45:53
last ntp sync: Fri Nov 29 16:45:29 2019
If the NTP server is not reachable, change it to a different NTP server and verify afterward if the time got synced properly.
Custom NTP server example :
config sys ntp
set type custom
config netserver
edit 1
set server pool.ntp.org
end
end
To list the current NTP config run:
get system ntp
show full system ntp
To verify if the NTP service is running verify if this command returns a process ID (PID):
diagnose sys process pidof ntpd
If no process ID is returned the process is not running.
This can be double-checked with the ps command which should show a process named 'ntpd':
fnsysctl ps
To verify if an implicit firewall policy got added to accept remote NTP requests use the iprope commands:
diagnose firewall iprope list | grep -f 123 -B11 -A1
diagnose firewall iprope list
To list open NTP sessions on port 123 run:
diagnose sys session filter clear
diagnose sys session filter dport 123
diagnose sys session list
diagnose sys session filter clear
Use the following command to create a network capture to verify if NTP packets are sent and received from the device:
diag sniffer packet any 'port 123' 6 0 l
To end the network capture after testing hit keys 'CTRL+C'.
Packet capture sample:
2024-07-30 02:28:30.375912 port1 -- 10.58.6.202.123 -> 10.58.6.72.123: udp 48
0x0000 704c a5fd e0f4 0069 6f6e ae01 0800 4500 pL.....ion....E.
0x0010 004c 0dcf 0000 8011 0b4d 0a3a 06ca 0a3a .L.......M.:...:
0x0020 0648 007b 007b 0038 d081 db00 0afa 0000 .H.{.{.8........
0x0030 3f76 0008 d1cf 0000 0000 ea53 2f0e 61a9 ?v.........S/.a.
0x0040 fbe7 0000 0000 0000 0000 0000 0000 0000 ................
0x0050 0000 ea53 30bd e999 9999 ...S0.....
2024-07-30 02:28:30.375979 port1 -- 10.58.6.72.123 -> 10.58.6.202.123: udp 48
0x0000 0069 6f6e ae01 704c a5fd e0f4 0800 4500 .ion..pL......E.
0x0010 004c 9f0a 0000 4011 ba11 0a3a 0648 0a3a .L....@....:.H.:
0x0020 06ca 007b 007b 0038 fd43 1c04 0afa 0000 ...{.{.8.C......
0x0030 3744 0000 0340 0a65 1414 ea53 2fe5 f719 7D...@.e...S/...
0x0040 9165 ea53 30bd e999 9999 ea53 30be 603e .e.S0......S0.`>
0x0050 d959 ea53 30be 603f 4eca .Y.S0.`?N.
If traffic is leaving out from the incorrect port, specify the interface under NTP configuration (For Example wan). The traffic should be left out from the correct interface.:
config system ntp
set interface "wan"
end
If the issue persists, continue to the next steps:
To enable debug logging for the ntpd process on the application layer run:
diagnose debug reset
diagnose debug disable
diagnose debug console timestamp enable
diagnose debug application ntpd -1
Debug sample output:
receive(10.58.6.202)
handle_client_message:977 from 10.58.6.202 vfid=0
Reply to 10.58.6.202.
waiting for 4 seconds ...
receive(10.58.6.202)
handle_client_message:977 from 10.58.6.202 vfid=0
Reply to 10.58.6.202.
Debug logging can be disabled after collecting the data with commands:
diagnose debug reset
diagnose debug disable
In case the NTPD process has a high CPU usage or a higher memory usage collect the following outputs while the issue is present:
First, find the PID of the NTP process.
diagnose sys process pidof ntpd
Then dump details about the process IDs:
diagnose sys process pstack <PID>
diagnose sys process dump <PID>
fnsysctl ls -al /proc/<PID>
fnsysctl cat /proc/<PID>/status
fnsysctl cat /proc/<PID>/stack
fnsysctl cat /proc/<PID>/limits
fnsysctl cat /proc/<PID>/maps
fnsysctl ls -al /proc/<PID>/fd
In case of a high CPU usage terminate the NTPD process and create a backtrace. This backtrace can provide details about the function in which the process got stuck:
diag sys kill 11 <PID>
diag debug crashlog read
Where <PID> is the process ID previously found with the command 'diagnose sys process pidof ntpd'.
It is also possible to keep track of the NTP status regularly using the API.
https://x.x.x.x/api/v2/monitor/system/ntp/status/?access_token='YourApiTokenHere'
This will provide similar data to 'diag sys ntp status' output which can be seen below:
Use the following link to configure Custom NTP Server .
