Description
This article describes a guideline and commands to troubleshoot any NTP synchronization issue on FortiGate and FortiSwitch devices.
Scope
FortiGate, FortiSwitch.
Solution
If the FortiGate is not able to sync the time with the configured NTP server, use the following commands to check the NTP server status:
get system stat
execute date
execute time
diagnose sys ntp status
An example output of the NTP status command is seen below:
diagnose sys ntp status
synchronized: yes, ntpsync: enabled, server-mode: disabled
ipv4 server(time.google.com) 216.239.35.0 -- reachable(0xff) S:1 T:5
server-version=4, stratum=1
reference time is e18b5929.fc81eb59 -- UTC Fri Nov 29 08:45:29 2019
clock offset is 0.011700 sec, root delay is 0.000000 sec
root dispersion is 0.000183 sec, peer dispersion is 5 msec
ipv4 server(time.google.com) 216.239.35.4 -- reachable(0xff) S:1 T:5 selected
server-version=4, stratum=1
reference time is e18b5929.c0f08d85 -- UTC Fri Nov 29 08:45:29 2019
clock offset is 0.009796 sec, root delay is 0.000000 sec
root dispersion is 0.000214 sec, peer dispersion is 8 msec
ipv4 server(time.google.com) 216.239.35.12 -- reachable(0xff) S:1 T:5
server-version=4, stratum=1
reference time is e18b592a.21f2218 -- UTC Fri Nov 29 08:45:30 2019
clock offset is 0.009603 sec, root delay is 0.000000 sec
root dispersion is 0.000214 sec, peer dispersion is 5 msec
ipv6 server(time.google.com) 2001:4860:4806:8:: -- reachable(0xfe) S:0 T:4
no data
ipv6 server(time.google.com) 2001:4860:4806:4:: -- reachable(0xfe) S:0 T:4
no data
ipv6 server(time.google.com) 2001:4860:4806:: -- reachable(0xfe) S:0 T:4
no data
ipv6 server(time.google.com) 2001:4860:4806:c:: -- reachable(0xfe) S:0 T:4
no data
ipv4 server(time.google.com) 216.239.35.8 -- reachable(0xff) S:1 T:5
server-version=4, stratum=1
reference time is e18b592a.bcbcf -- UTC Fri Nov 29 08:45:30 2019
clock offset is 0.013359 sec, root delay is 0.000000 sec
root dispersion is 0.000198 sec, peer dispersion is 3 msec
execute time
current time is: 16:45:53
last ntp sync: Fri Nov 29 16:45:29 2019
If the NTP server is not reachable, change it to a different NTP server and verify afterward if the time has synced properly.
By default, FortiGate will take the NTP server from FortiGuard. For custom NTP server configuration, it is necessary to enable the user option.
The custom NTP server configuration option will be greyed out. This option can only be enabled via CLI.
Custom NTP server example:
config system ntp
set type custom
config ntpserver
edit 1
set server pool.ntp.org
end
end
To list the current NTP config, run:
get system ntp
show full system ntp
dns-txt-fgt # get system ntp
ntpsync : enable
type : custom
syncinterval : 60
ntpserver:
== [ 1 ]
id: 1
source-ip : 0.0.0.0
source-ip6 : ::
server-mode : enable
authentication : disable
interface : "fortilink"
dns-txt-fgt # show full-configuration system ntp
config system ntp
set ntpsync enable
set type custom
set syncinterval 60
config ntpserver
edit 1
set server "pool.ntp.org"
set ntpv3 disable
set authentication disable
set interface-select-method auto
next
end
set source-ip 0.0.0.0
set source-ip6 ::
set server-mode enable
set authentication disable
set interface "fortilink"
end
To verify if the NTP service is running, verify if this command returns a process ID (PID):
diagnose sys process pidof ntpd
If no process ID is returned, the process is not running.
This can be double-checked with the ps command, which should show a process named 'ntpd':
fnsysctl ps
Super Admin privilege is required to run the 'fnsysctl' command. Otherwise, FortiGate will return an error, as explained in Troubleshooting Tip: fnsysctl command returns Unknown action 0
To verify if an implicit firewall policy got added to accept remote NTP requests, use the iprope commands:
diagnose firewall iprope list | grep -f 123 -B11 -A1
diagnose firewall iprope list
To list open NTP sessions on port 123, run:
diagnose sys session filter clear
diagnose sys session filter dport 123
diagnose sys session list
diagnose sys session filter clear
Use the following command to create a network capture to verify if NTP packets are sent and received from the device:
diagnose sniffer packet any 'port 123' 6 0 l
To end the network capture after testing, hit the keys 'CTRL+C'.
Packet capture sample:
2024-07-30 02:28:30.375912 port1 -- 10.58.6.202.123 -> 10.58.6.72.123: udp 48
0x0000 704c a5fd e0f4 0069 6f6e ae01 0800 4500 pL.....ion....E.
0x0010 004c 0dcf 0000 8011 0b4d 0a3a 06ca 0a3a .L.......M.:...:
0x0020 0648 007b 007b 0038 d081 db00 0afa 0000 .H.{.{.8........
0x0030 3f76 0008 d1cf 0000 0000 ea53 2f0e 61a9 ?v.........S/.a.
0x0040 fbe7 0000 0000 0000 0000 0000 0000 0000 ................
0x0050 0000 ea53 30bd e999 9999 ...S0.....
2024-07-30 02:28:30.375979 port1 -- 10.58.6.72.123 -> 10.58.6.202.123: udp 48
0x0000 0069 6f6e ae01 704c a5fd e0f4 0800 4500 .ion..pL......E.
0x0010 004c 9f0a 0000 4011 ba11 0a3a 0648 0a3a .L....@....:.H.:
0x0020 06ca 007b 007b 0038 fd43 1c04 0afa 0000 ...{.{.8.C......
0x0030 3744 0000 0340 0a65 1414 ea53 2fe5 f719 7D...@.e...S/...
0x0040 9165 ea53 30bd e999 9999 ea53 30be 603e .e.S0......S0.`>
0x0050 d959 ea53 30be 603f 4eca .Y.S0.`?N.
If traffic is leaving out of the incorrect port, specify the interface under NTP configuration (for example, WAN). The traffic should be left out of the correct interface:
config system ntp
set interface "wan"
end
If the issue persists, continue to the next steps:
To enable debug logging for the NTPD process on the application layer, run:
diagnose debug reset
diagnose debug disable
diagnose debug console timestamp enable
diagnose debug application ntpd -1
diagnose debug enable
Debug sample output:
receive(10.58.6.202)
handle_client_message:977 from 10.58.6.202 vfid=0
Reply to 10.58.6.202.
waiting for 4 seconds ...
receive(10.58.6.202)
handle_client_message:977 from 10.58.6.202 vfid=0
Reply to 10.58.6.202.
Debug logging can be disabled after collecting the data with commands:
diagnose debug reset
diagnose debug disable
In case the NTPD process has a high CPU usage or a higher memory usage, collect the following outputs while the issue is present:
Find the PID of the NTP process.
diagnose sys process pidof ntpd
Dump details about the process IDs:
diagnose sys process pstack <PID>
diagnose sys process dump <PID>
fnsysctl ls -al /proc/<PID>
fnsysctl cat /proc/<PID>/status
fnsysctl cat /proc/<PID>/stack
fnsysctl cat /proc/<PID>/limits
fnsysctl cat /proc/<PID>/maps
fnsysctl ls -al /proc/<PID>/fd
In case of a high CPU usage, terminate the NTPD process and create a backtrace. This backtrace can provide details about the function in which the process got stuck:
diagnose sys kill 11 <PID>
diagnose debug crashlog read
Where <PID> is the process ID previously found with the command 'diagnose sys process pidof ntpd'.
It is also possible to keep track of the NTP status regularly using the API.
https://x.x.x.x/api/v2/monitor/system/ntp/status/?access_token='YourApiTokenHere'
This will provide similar data to the 'diagnose sys ntp status' output, which can be seen below:
Related articles:
Technical Tip: Custom NTP server configuration
