FG300E restarts about every 3 minutes

[j_such]

Hello,

I put new configured FG300E HA with FortiOS 5.4.7 into production network and this FG started to reboot about every 3 minutes and few seconds.

Crashlog gives no information about source.

How to debug this case ?

I don't want to move to FortiOS 5.6.x because many older FortiAPs will not be supported. I'm afraid to move to FortiOS 5.4.8 because of known bug: 470399 The FG-300E/301E and FG-500E/501E reboots with kernel panic errors

Perhaps downgrade to FortiOS < 5.4.6 ?

A few days ago I've made ticket in Fortinet Support but no helpful suggestions received ...

Janusz Such

[Toshi_Esumi · Accepted solution]

First, you should call in the number in your country/region. You might need to be in queue some time but hopefully not long. Since you already open a TT via support web site, you don't have to get it created by the first rep before you're placed in a queue. You must have attached config and other log data so it would be shorter for the tech who you grabbed to get to the speed.

But likely need to flush the boot drive and reload the image either 5.4.7 or previous version you upgraded it from via TFTP, and upload the saved config of the version. Then you can upgrade to 5.4.8.

[yoda · Accepted solution]

Hello,

this kind of FG300E rebooting might be linked to the usage of WiFis provided by FortiAPs managed by the FG300E. In our case after upgrade to FortiOS 5.6.3 no more such rebooting observed. If upgrade to FortiOS 5.6.3 is not an option then you might manage your FortiAPs by a separate FortiGate (connected to the FG300E).

Yoda

[tanr · Accepted solution]

Have you hooked up a laptop directly to the console (RJ45 serial) port to watch the whole sequence?

That should show you all output during the reboot, much of which you can miss if you're just using SSH or HTTPS to access it.

[ede_pfau · Accepted solution]

About the pings: they are most probably not blocked but just 'vanish from sight'.

Typically, after a new session is established, only 2 pings are forwarded by the CPU and then the session is offloaded to the NP ASIC. The built-in sniffer cannot display ASIC traffic and so it seems that traffic is blocked. But it's only offloaded and invisible.

[Toshi_Esumi · Accepted solution]

First, you should call in the number in your country/region. You might need to be in queue some time but hopefully not long. Since you already open a TT via support web site, you don't have to get it created by the first rep before you're placed in a queue. You must have attached config and other log data so it would be shorter for the tech who you grabbed to get to the speed.

But likely need to flush the boot drive and reload the image either 5.4.7 or previous version you upgraded it from via TFTP, and upload the saved config of the version. Then you can upgrade to 5.4.8.

[yoda · Accepted solution]

Hello,

this kind of FG300E rebooting might be linked to the usage of WiFis provided by FortiAPs managed by the FG300E. In our case after upgrade to FortiOS 5.6.3 no more such rebooting observed. If upgrade to FortiOS 5.6.3 is not an option then you might manage your FortiAPs by a separate FortiGate (connected to the FG300E).

Yoda

[tanr · Accepted solution]

Have you hooked up a laptop directly to the console (RJ45 serial) port to watch the whole sequence?

That should show you all output during the reboot, much of which you can miss if you're just using SSH or HTTPS to access it.

[j_such]

I received [ from support ] pre-release version of 5.4.9 and it works better.

Thank you for your post !

[j_such]

Hello Yoda,

I also suspected that these reboots are WiFi related.

I received [ from support ] pre-release version of 5.4.9 and it works better, no further reboots.

Another problem arose that in many firewall policies [ Wifi related ] I had to turn off hardware acceleration by NP by command:

set auto-asic-offload enable

without this change e.g. two pings to another VLAN are OK and subsequent are blocked.

I noticed about it Fortinet Support.

Your idea about managing FortiAPs by separate FortiGates [ we have 2 x FG310B which will be replaced by 2 x FG300E ] is interesting and worth considering.

Thank you for your post !

[ede_pfau · Accepted solution]

About the pings: they are most probably not blocked but just 'vanish from sight'.

Typically, after a new session is established, only 2 pings are forwarded by the CPU and then the session is offloaded to the NP ASIC. The built-in sniffer cannot display ASIC traffic and so it seems that traffic is blocked. But it's only offloaded and invisible.

[j_such]

Hello ede_pfau

I understand that traffic offloaded from CPU to NPU vanish from debug flow but in my case traffic was really blocked and no answer for third, fourth, etc ping was received. Also other traffic e.g. RDP connections was blocked.

Perhaps a bug in pre-release version of 5.4.9 ...

[Toshi_Esumi]

If you were running sniffing at the destination device and saw only two ICMP request packets arrived, I have to accept your claim. But since you already got the pre-release from TAC they must know something specifically with 300E and they would tell what might be going on.

[yoda · Accepted solution]

Hello! In the meantime (as mentioned earlier) the FG300E is on FW 5.6.3. I try to remember how I managed it with 5.4.7 and a separate firewall controlling the FortiAPs: So we had two firewalls (both with FW 5.4.7): - FW1 : the FG300E providing the network interface to which the FortiAPs are connected and located in a productive network environment on separate floors (separate network switches in use) - FW2 : another firewall (in our case a FG100D) acting as WiFi controller only On FW1 : On the network interface where FortiAPs are connected to a new VLAN with DHCP enabled (for administration of FortiAP connections/policies) has to be defined, example : name "Mgmt_FAP", VLAN ID 50, IP 10.50.11.1/24, DHCP server enabled, HTTPS,PING,CAPWAP,SSH enabled A link between both firewalls (connected via physical network patch cable) using a /30 linknet has to be established, port x on FW1 and port y on FW2 are both parts of this common /30 linknet , CAPWAP to be enabled on involved interfaces. examples for interfaces used for the link network : FW1, port x : name "lk2fg100d" , IP 172.16.100.1/30 FW2, port y : name "lk2fg300e",  IP 172.16.100.2/30 On each of these new interfaces a separate VLAN for each WiFi needed has to defined no CAPWAP, no DHCP server, role LAN, examples: FW1 : interface name: "lk2fg100d_vl71", VLAN ID 71, IP 172.16.71.1/30, FW2 : interface name: "lk2fg300e_vl71", VLAN ID 71 , IP 172.16.71.2/30 On FW2 a default route to interface "lk2fg300e", gateway IP : link network IP1 of FW1 (in our example 172.16.100.1 ) has to be defined On FW1 services "CAPWAP": UDP5246,5247 has to be defined and a FW policy for traffic from "Mgmt_FAP" port to "lk2fg100d" and services "CAPWAP" is needed. Again as an example a WiFi on FW2 has to be defined as follows: -    SSID: "FAP_example_71" with  IP 172.16.171.1/24, DHCP server enabled, propagating SSID:"example_71",     default settings for DNS, NTP, etc. as you prefer -    Policy routes for traffic from WiFi to Link-interface of FW1,     example : incoming interface: FAP_example_71, src adr:172.16.171.0/24,

                   outgoing interface: lk2fg300e_vl71, gw addr:172.16.71.1 -    FW policies for traffic from WiFi to Link-interface of FW1,     example: incoming interface: FAP_example_71, outgoing interface:lk2fg300e_vl71, NAT enabled Also needed on FW1: -    Static routes for (retour) traffic to WiFis on FW2,       example : destination:172.16.171.0/24, device:lk2fg100d_vl71, gateway:172.16.71.2 -    Various policies from VLANs initiated at FW2 to various targets on interfaces on FG1 interface,       example: lk2fg100d_vlan71 to [WAN and other interfaces], NAT enabled The FortiAPs should be configured via separate management VLAN and controlled by FW2. As an example please have a look onto according FortiAP configuration commands like cfg -a AP_MGMT_VLAN_ID=50 cfg -a AC_IPADDR_1=172.16.100.2 Yoda