Troubleshooting Tip: Firewall stopped responding

[salemneaz]

Description	This article gives an overview for the action that can be applied in the event a FortiGate becomes unresponsive.
Scope	FortiOS.
Solution	If the FortiGate firewall is not responding, the following steps can be taken to diagnose the issue. Check the LED Status: First, check if the LEDs are active or not. Depending on the model, the Power LED is located either on the front panel, or at the back of the unit, next to each PowerSupply installed. If the LED is lit up green, that means that the Power is okay. Check the Connectivity: The next step is to establish connection to the FortiGate. Either via GUI/SSH, or via console connection. Ping the FortiGate Internal LAN or the management IP from the computer. Sometimes, the Ping might be enabled at the allowed access. If so, try to access the FortiGate using the GUI, or SSH into it. Open https://<FortiGate-IP> in a browser. Try connecting via SSH using ssh admin@<FortiGate-IP>  C:\Users\sneaz>ssh admin@10.0.0.11 The authenticity of host '10.0.0.11 (10.0.0.11)' can't be established. ED25519 key fingerprint is SHA256:xxxxxxxxxxxxxxxxxxxxxxx. This key is not known by any other names. To continue connecting (yes/no/[fingerprint])? yes Warning: Permanently added '10.0.0.11' (ED25519) to the list of known hosts. admin@10.0.0.11's password: FGT-1 # Note: The message 'The authenticity of host '10.0.0.11 (10.0.0.11)' can't be established' appears if the computer does not have the fingerprint saved. After saving the fingerprint, this message will not appear anymore. Confirm the cables are connected properly. Check for IP address conflicts via the Windows event logs. Verify reach to the internet: Execute a ping to the internet with the following commands: exec ping 8.8.8.8 PING 8.8.8.8 (8.8.8.8): 56 data bytes 64 bytes from 8.8.8.8: icmp_seq=0 ttl=117 time=14.3 ms 64 bytes from 8.8.8.8: icmp_seq=1 ttl=117 time=14.8 ms 64 bytes from 8.8.8.8: icmp_seq=2 ttl=117 time=14.5 ms 64 bytes from 8.8.8.8: icmp_seq=3 ttl=117 time=16.2 ms 64 bytes from 8.8.8.8: icmp_seq=4 ttl=117 time=13.5 ms --- 8.8.8.8 ping statistics --- 5 packets transmitted, 5 packets received, 0% packet loss round-trip min/avg/max = 13.5/14.6/16.2 ms exec ping www.google.com      PING www.google.com (142.250.191.164): 56 data bytes 64 bytes from 142.250.191.164: icmp_seq=0 ttl=116 time=45.0 ms 64 bytes from 142.250.191.164: icmp_seq=1 ttl=116 time=40.6 ms 64 bytes from 142.250.191.164: icmp_seq=2 ttl=116 time=46.0 ms 64 bytes from 142.250.191.164: icmp_seq=3 ttl=116 time=47.3 ms 64 bytes from 142.250.191.164: icmp_seq=4 ttl=116 time=45.8 ms --- www.google.com ping statistics --- 5 packets transmitted, 5 packets received, 0% packet loss round-trip min/avg/max = 40.6/44.9/47.3 ms Check the routing table to make sure that the default route is installed at the routing table get router info routing-table all Codes: K - kernel, C - connected, S - static, R - RIP, B - BGP O - OSPF, IA - OSPF inter area N1 - OSPF NSSA external type 1, N2 - OSPF NSSA external type 2 E1 - OSPF external type 1, E2 - OSPF external type 2 i - IS-IS, L1 - IS-IS level-1, L2 - IS-IS level-2, ia - IS-IS inter area V - BGP VPNv4 * - candidate default Routing table for VRF=0 S* 0.0.0.0/0 [4/0] via 10.0.0.1, port1, [1/0] <----- Default route is present at the table. C 10.0.0.0/24 is directly connected, port1 S 172.16.20.0/24 [254/0] is a summary, Null, [1/0] C 172.17.10.1/32 is directly connected, Loopback-1 C 192.168.10.0/24 is directly connected, port10 Check the Console Port: Use a console cable to connect with the FortiGate. Technical Tip: How to connect to the FortiGate and FortiAP console port After connecting, check if the login prompt is displayed, or some errors appear.  When some errors or text is continuously displayed, there are a few possible scenarios: The unit may be stuck in a boot loop. If the error is not clear, log all of these errors in the terminal and provide them to TAC to diagnose the problem. Another possible issue is that FortiOS may be hung with different error messages. These are two examples: Fatal error: Loading FOS fails!   power cycle. System halted. Fatal error: AV engine file authentication failed!   power cycle. System halted.  If these logs are seen in the firewall CLI, FortiGate needs to be formatted and new firmware needs to be loaded using the TFTP server (Technical Tip: Installing firmware from system reboot) The unit does not display anything on the console port. In this case, the message is likely related to scenario #2, but the message was already displayed but not captured in the console. In this case, press the NMI button (if available on the specific hardware - Technical Tip: NMI Button for Troubleshooting Kernel Issues) to send the kernel crash to the console. If this button is not available, no further logs can be obtained from the unit in this state, so keep the console connection running and power-cycle the unit. The boot sequence and possible errors should be now visible. If the login prompt is seen, log in and check the available crash logs ('diag debug crashlog read'), comlog (if available, and enabled ('diag debug comlog read'), as well as the system status ('get system status') and the performance status ('get system perf status'). For the performance status, check if SoftIrqs are incrementing. See Troubleshooting Tip: Check SoftIrq increments (recommended when experiencing high CPU usage). Related articles: Troubleshooting Tip: Check SoftIrq increments (recommended when experiencing high CPU usage)  Troubleshooting Tip: Unable to boot the firewall or load firmware image