Missing Firewall Policies after Firmware Upgrade or Restarts

[hmiranda]

Hi all!

A department on the company that I work are having issues with some FortiGates. This department is charge to the maintenance of the client's fortigates. Today they had two issues with "missing policies", one with a FG200D and another with a FortiWiFi 90D. The FG200D is installed on our office and it had "missing policies" after a Firmware upgrade following the upgrade path from FortiOS 5.6.3 to 5.6.6 and finally 6.0.3. The other device is a FortiWiFi 90D owned by one of our clients and this one also had "missing policies", but in this case it happened after a power disruption.

We had this type of issue on the past on other devices, but were few. The department of maintenance tells me that they are following the recommendations of FortiNet for upgrades by reading the release notes and following the upgrade paths. My concern is that this is happening more often and we are scheduling to do a firmware upgrade to a core 1000D that manage the security on our DataCenter. Missed policies on this FortiGate will cause me a huge stress.

I had been looking on the forum if someone else had this problem, but I didn't find nothig. Some one faced or is facing with this "missing policies" issue? Do they are missing something? Can they do something to mitigate this? I had also hear them say that "policies are getting corrupted" after an upgrade.

Regards,

Harry

[Dave_Hall · Accepted solution]

This hasn't happen with recent firmwares, but back in the 4.2.x days, the "conversion scripts" that ran during the firmware upgrade process didn't properly handle white space characters in object names/labels, would truncate them at the first space character, so it pretty much missed up the config rather badly. 

These days, I make it a habit to run the unencypted backup of the configs (before and after the firmware upgrade) to see what has changed.  

I suggest doing the above, as well, run "diagnose debug config-error-log read" at the CLI following the firmware upgrade.

[emnoc · Accepted solution]

Same here, never seen that issues b4. I would suspect that maybe  manual  save might be a issue. If you login into the Fortigate and look and diff the rev b4 and after the changes, what that  system revision shows ?  FWIW 

In big  firewall deployments I've always make a backup, and  save the cfg, and check the revisions prior to any changes and run the webgui rev-diff

Ken Felix

[rwpatterson · Accepted solution]

Way back when dirt was invented, Fortigates would sometime run into memory issues if run for too long. If the device has been up for a while, try rebooting it prior to starting the upgrade. That adds a few minutes to the procedure, but may alleviate some issues.

[Dave_Hall · Accepted solution]

This hasn't happen with recent firmwares, but back in the 4.2.x days, the "conversion scripts" that ran during the firmware upgrade process didn't properly handle white space characters in object names/labels, would truncate them at the first space character, so it pretty much missed up the config rather badly. 

These days, I make it a habit to run the unencypted backup of the configs (before and after the firmware upgrade) to see what has changed.  

I suggest doing the above, as well, run "diagnose debug config-error-log read" at the CLI following the firmware upgrade.

[hmiranda]

Hi Dave,

They have a habit of putting spaces in the names of objects. In the FortiGate of our office I just executed the command "diagnose debug config-error-log read", but showed nothing. I do not know if it's because the upgrade was a few hours ago. 

Thank you for answering. I will pass your suggestion to them.

[emnoc · Accepted solution]

Same here, never seen that issues b4. I would suspect that maybe  manual  save might be a issue. If you login into the Fortigate and look and diff the rev b4 and after the changes, what that  system revision shows ?  FWIW 

In big  firewall deployments I've always make a backup, and  save the cfg, and check the revisions prior to any changes and run the webgui rev-diff

Ken Felix

[rwpatterson · Accepted solution]

Way back when dirt was invented, Fortigates would sometime run into memory issues if run for too long. If the device has been up for a while, try rebooting it prior to starting the upgrade. That adds a few minutes to the procedure, but may alleviate some issues.

[hmiranda · Accepted solution]

emnoc wrote:

Same here, never seen that issues b4. I would suspect that maybe  manual  save might be a issue. If you login into the Fortigate and look and diff the rev b4 and after the changes, what that  system revision shows ?  FWIW 

In big  firewall deployments I've always make a backup, and  save the cfg, and check the revisions prior to any changes and run the webgui rev-diff

Ken Felix

Ken,

I did the rev-diff comparing two backup files. The first two were the last backup on version 5.6.3 and the first (and only) on 5.6.6. This presented that a lot of policies were deleted. This doesn't happened when I did the second comparison between the last backup of 5.6.3 and the first of 6.0.3. This last comparison showed policies that the first comparison showed them as deleted. The last comparison (5.6.3 and 6.0.3) showed clearly that the policies that affected the services attached to their interfaces were no longer on 6.0.3.

Thanks for your suggestions about the tasks for before and after upgrading a device.