FortiOS 5.4.1 Issues

[Pacolo]

Hello everyone,

I have a cluster of 1500D in A-P mode working and I have noticed some rare issues since we upgraded to 5.4.1.

For example, after disable and enable again a policy route rule, on the GUI it was placed as SEQ #1, but when running a "diagnose firewall proute list" it was placed at the end of the policy rules.

Another one, a policy fw rule created on FortiOS 5.2.x (rule 41) doest not match traffic that is supposed to, another rule with the same config created on FortiOS 5.4.1 (rule 121) indeed match the same traffic.

Taking a view at the rule's full-config, the change of the new rule created in FortiOS 5.4.1 is "set utm-status enable" instead of "set utm-status disable". Although we have no utm feature enabled, it adds the following sentences:

set profile-type single set av-profile '' set webfilter-profile '' set dnsfilter-profile '' set spamfilter-profile '' set dlp-sensor '' set ips-sensor '' set application-list '' set casi-profile '' set voip-profile '' set icap-profile '' set waf-profile '' set profile-protocol-options '' set ssl-ssh-profile ''

Anyway, I changed to "set utm-status enable", and it did not match the traffic neither, so now I do not trust which rules created on FortiOS 5.2 are working on 5.4.

Other isssue would be that on another cluster of 1000D, the Virtual Wire config was lost after a reboot.

I am going to open a case about the rules's issues as they have given us a some headache, but I think we should downgrade to the 5.2 branch until the 5.4 is more stable.

Has anybody found similar issues?

Regards,

Paco.

[emnoc]

For example, after disable and enable again a policy route rule, on the GUI it was placed as SEQ #1, but when running a "diagnose firewall proute list" it was placed at the end of the policy rules.  

That's normal behavior for all FortiOS. The route ID# is not a "SEQ" number as you indicated. It's a route ID and in the RIB it's still the exact match that wins regardless if it  a PBR or static-Route. Every route you populate in the   PBR will be placed at the end of the list regardless of the ID number you give it.

You can check the  table via "get router  info routing-table  database" or the rt-cache  "

diag ip rtcache  list" and monitor the use field for the  route that was selected 

[tanr]

Paco, after your changes to the rule that had been converted from 5.2 to 5.4.1, what are the differences between the two rules full-configs?  

Also, what is the rule supposed to be matching, exactly?  Have you confirmed which other rule (like the default deny) is actually matching the traffic?

So far, with 5.4.1 I've run into two issues matching security policies:

[ul]

The (documented) issue of a proxy-based Web Filter not catching what it should when using a flow-mode only Application Control profile.  Changing the Web Filter to a flow-mode version worked around this.

A non-reproducible issue where a security policy set to match Device Types (iPhone) failed to match devices of type iPhone that had been given an alias.  I saw this happen very obviously, then modified the rule and re-ran tests of this and was never able to reproduce it again.  This had happened after I edited the policy in the GUI directly from the list (clicking on the source subnet, then using the right pane to add device types) so its possible the issue is related to that sort of editing.  But as I said, couldn't repro it later.[/ul]

If you got a reproducible difference in the way the rule gets handled this seems like a good issue to open a support ticket about.  

[tanr]

Spoke to soon.  I re-checked recent logs and found a repro of my second case where a security policy set to match a Device Type (iPhone) fails to match devices of type iPhone that have been given an alias, thus making them a custom device (though they still have type iPhone).

When this occurs none of my security policies match (even my last any:any/all:all deny) and it drops to the implicit deny (rule id 0).  What is very weird is that this only seems to happen for a specific service that is a UDP range (Apple RTP RTCP) that the default IOS outbound security policy would have allowed.  The same custom device, with other services, is matching the rule and iPhone device type as it should for everything except that service.

Looks like I'll be opening a ticket.

[tanr]

Did some more research on this, and it looks more like an iOS 10 bug.

The iOS 10 devices are attempting UDP 16403, not to some external wan IP, but to my own public IP (the FortiGate's public IP).  The FortiGate is blocking them directly (through local-in-policy handling?), which I assume is why I'm seeing these under the implicit deny.

[Pacolo]

tanr, thanks for the sharing your view, though I am afraid that I haven't worked with the device identification feature, so I do not know what kind of problems may appear with it.

The difference between the full-config on the rules I tested is the set utm-status enable/disable.

Yesterday we had another issue with the new feature introduced in FortiOS 5.4, DNS Filter, which help to block Botnets via DNS, categorize the DNS requests or block the DNS requests via a local list of domains, to block access to HTTPS sites without the deep inspection feature.

This was the most critical issue, as it caused a network outage because it appears that these requests done to Fortiguard servers started to fail.

On the WebFilter log...

"msg="A rating error occurs" error="all Fortiguard SDNS servers failed to respond"

...meanwhile the customer traffic was flowing (around 3 Gbps), there were a lot of DNS requests 400K, which in a normal situation were 150K, so it appears that there were a lot of retries, and the responses were slow, but on a certain moment, it dropped all of them.

Our local sales engineer has suggested us to rollback to the 5.2 branch, as there are a lot of bugs on this new branch.

I have opened a case for them to work on these issues in case they can fix them on future releases.

Regards,

Paco.

[tanr]

For each DNS Filter profile, you can turn on the option for "Allow DNS requests when a rating error occurs" which allows DNS requests through if the Fortinet servers aren't available.

Per interface, you can set the option "Scan Outgoing Connections to Botnet Sites" to Block.  Having that set checks against an already downloaded list of Botnet IP's, so it's separate from the DNS Filter Botnet check.  That's only if you have paid for the Mobile and Botnet FortiGuard service, though.

[Jeremiah_Jackson]

I feel like there is a memory leak in the new version.  In 5.2 we never hit conserve mode, but I am having trouble avoiding it in the new version.

[Pacolo]

 Hello tanr,

That's right, I checked the option "Allow DNS requests when a rating error occurs", though after that, the DNS requests still returned the errors, so it did not worked.

Regarding the Botnet feature, I have looked for the info again and it is not very clear, as on the Handbook, it says that "This database is covered by FortiGuard web filter licensing, so you must have a FortiGuard web filtering license to use this feature." 

Although on the What's new section of the Handbook...

"Access to the IRDB has been available to users through FortiCare support contracts. This will change and users

will have to subscribe to the IRDB either through the FortiGuard Mobility Security Service (FMSS) or the FortiGuard Enterprise Bundle."

This is new, it is not clear with the "This will change" statement, furthermore it was not mentioned on the Release Notes of the FortiOS 5.4.1, very bad Fortinet.

Anyway, I can see Botnet IPs on the IRDB, but not domains on the BDDB, but the actual problem was that the Forti servers were returning errors and not working as expected.

[Pacolo]

Hi Jeremiah,

The strange thing is that even though the device reached 93% of Memory Usage, it did not trigger the Conserver Mode.

(global) # diag hardware sysinfo shm SHM counter: 406227 SHM allocated: 6782744 SHM total: 15952420864 conserve mode: off system last entered: n/a sys fd last entered: n/a SHM FS total: 16281268224 SHM FS free: 16269312000 SHM FS avail: 16269312000 SHM FS alloc: 11956224

(global) # get system performance status  ... Memory states: 86% used

At the moment, 86% and no conserve mode.