Fortigate 60E - dropping internet connection

[pgregor]

Hi, we have Fortigate 60E.

Last two months we have problem with unexpected breaking of 60E operation.

It breaks all traffic for few minutes and after that it starts operating (no admin action is required).

We have small office with up to 15 PCs and up to 20 virtual servers within our intranet. 

We have 2 internet connections and there is no problem with ISPs.

Both lines are stable with no outages.

The first internet line is about 20Mbit the second one is 5Mbit. 

The problem with Fortigate occures only if we use first (20Mbit) line.

If I disconnect WAN during the time of Fortigate outage and immediatelly I connect it back to Fortigate,

the Fortigate outgage is solved.

Thanks for any idea.

Petr

[Dave_Hall]

Have you configured the two ISP connections under SD-WAN?  Have you set the ingress/egress values on both ISP connections? Does the Bandwidth history graph for the 20Mbit line show the connection being maxed out (what about the 5 Mbit line)?  What does FortiView shows which devices are using up most of the bandwidth?   Have you checked for duplex/speed mismatch or line cable issues?  (e.g. perform diag hardware deviceinfo nic <interface name> on the CLI and check for errors - perform the diag test again in a few mins if there are errors and see if the counters increase.)

[emnoc]

Also double check speed/duplex, just had this issue in a 60D where the ISP changed hardware and the FGT-nic would reset like every 9-12 mins. I had to lock the speed/duplex

Ken Felix

[pgregor]

The ISPs are under SD-WAN. The main is 20Mbit line. When it is down, then 5Mbit line is switched (and back).

There is no error in diag commands (wan1 is 5Mbit, wan2 20Mbit).

We used the second ISP because of the main line was often down.

When 5Mbit is used alone then no problem occurres.

How to lock the speed/duplex? On interface wan?

We have set Traffic Shapers only.

Petr

Info:

FG-Orsia # dia hardware deviceinfo nic wan1
Description :FortiASIC NP6LITE Adapter
Driver Name :FortiASIC NP6LITE Driver
Board :60E
lif id :0
lif oid :64
netdev oid :64
tx group :1
Current_HWaddr e8:1c:ba:75:f7:d2
Permanent_HWaddr e8:1c:ba:75:f7:d2
========== Link Status ==========
Admin :up
netdev status :up
autonego_setting:1
link_setting :1
speed_setting :10
duplex_setting :0
Speed :1000
Duplex :Full
link_status :Up
============ Counters ===========
Rx Pkts :16466947
Rx Bytes :14658704761
Tx Pkts :12558736
Tx Bytes :2655143521
Host Rx Pkts :7674970
Host Rx Bytes :6527220133
Host Tx Pkts :5883525
Host Tx Bytes :663989880
Host Tx dropped :0
 
FG-Orsia # dia hardware deviceinfo nic wan2
Description :FortiASIC NP6LITE Adapter
Driver Name :FortiASIC NP6LITE Driver
Board :60E
lif id :1
lif oid :65
netdev oid :65
tx group :2
Current_HWaddr e8:1c:ba:75:f7:d3
Permanent_HWaddr e8:1c:ba:75:f7:d3
========== Link Status ==========
Admin :up
netdev status :up
autonego_setting:1
link_setting :1
speed_setting :10
duplex_setting :0
Speed :1000
Duplex :Full
link_status :Up
============ Counters ===========
Rx Pkts :12498696
Rx Bytes :10747982776
Tx Pkts :9288245
Tx Bytes :1628554518
Host Rx Pkts :6966142
Host Rx Bytes :5388007857
Host Tx Pkts :4686829
Host Tx Bytes :759243061
Host Tx dropped :0

[Dave_Hall]

Unless someone can say otherwise, I do not think you need to worry about the duplex/speed as an issue the diag tests does not show there being a problem with them - otherwise  you would get various rx/tx counter errors, that would increase over time.  That said, if you want to set/force the duplex/speed on an interface, you can do this via the CLI:

config system interface     edit <interface name>         set speed ?     next end where ? is: auto        Automatically adjust speed. 10full      10M full-duplex. 10half      10M half-duplex. 100full     100M full-duplex. 100half     100M half-duplex. 1000full    1000M full-duplex. What I mean by setting the ingress/egress values on both ISP connections is to set values for "Estimated Bandwidth" on each Interface. 

Later fgt firmware versions come with some nice SD-WAN settings/monitoring tools.  I would make sure that the all WAN interfaces have the proper default route, distance/metric, and you have setup the load-balancing (aka SD-WAN Rules).  The SD-WAN monitor will tell you how many sessions are open/going out which ISP connection. 

If you do not have a bandwidth history graph on the main dashboard, I suggest adding two (one for each ISP connection). I would monitor the bandwidth usage, and CPU, memory, and sessions.  The fgt will (should) go into conserve mode should memory usage go near/over 80%. 

If you have direct access to the ISP gateway devices, I would log into each device and check for any log or events.  Sometimes one side of that WAN connection may look fine, but the other side may tell a different story.

If you have ping watch guard settings enabled (under Performance SLA) you will likely want to confirm they are working as expected.  If you are using Google's DNS there is rate limits set on how often you can ping their DNS servers.

And of course you should check the System Events/Router Events (under Log & Report) for issues.

[pgregor]

I set Estimated Bandwidth, but nothing changed.

Static Route is set for interface SD-WAN and Dynamic Gateway is enabled. Routing monitor shows correct settings.

I can not monitor the main ISP, I only ping to it.

I have enabled a bandwidth history graph on the main dashboard and sessions and so on, but everything looks fine.

System Events:

[ul]

when wan2 is down[/ul]

The member2(wan2) link is unreachable or miss threshold. Stop forwarding traffic.

Service1(VLAN-wan2) will failover to other available interface(s).

[ul]

when wan2 is up[/ul]

The member2(wan2) link is available. Start forwarding traffic.

Service1(VLAN-wan2) prioritized by latency will be redirected in seq-num order 2(wan2).

Current SD-WAN settings:

config system virtual-wan-link
    set status enable
    set load-balance-mode weight-based
    config members
        edit 2
            set interface "wan2"
            set gateway 212.158.144.193
        next
        edit 4
            set interface "wan1"
            set gateway 192.168.8.1
        next
    end
    config health-check
        edit "Google"
            set server "8.8.8.8"
            set interval 10
            set update-static-route disable
            set members 2 4
            config sla
                edit 1
                next
            end
        next
        edit "Quad9"
            set server "9.9.9.9"
            set interval 10
            set update-static-route disable
            set members 2 4
            config sla
                edit 1
                next
            end
        next
    end
    config service
        edit 1
            set name "VLAN-wan2"
            set mode priority
            set dst "all"
            set src "ORSIA-VLAN102-VoIP" "ORSIA-VLAN103-Guest" "ORSIA-VLAN104-DMZ" "ORSIA-VLAN199-MGMT" "ORSIA-VLAN101-LAN"
            set health-check "Google"
            set priority-members 2
        next
    end
end

The problem is on wan2 only. It does not matter additional wan1 is connected or not. Wan1 alone worked fine too.

Wan2 is down for minutes (last time yesterday 16 minutes, 27 minutes, 30 minutes), sometimes only for a while.

If I restart Fortigate, wan2 goes up.

[Dave_Hall]

If WAN2 goes down are you able to ping WAN2's GW address (or the ISP's modem/route device) from/through WAN1 connection and/or from another location on the Internet? (e.g. http://www.kloth.net/services/).  I am speculating the ping server settings in the health-check section may need to be tweaked (i.e. perhaps set the interval higher than 10.)

[pgregor]

If WAN2 goes down, WAN2's GW ping is not accessible (from intranet). From the Internet is not accessible our public IP which is set on WAN2. I've already tried to disable ping or set it to 10, no change. I think it could be caused by our ISP. For the past few days we have had twice the speed of wan2 (better line?) and there are no downs yet. We will see.

[Denver121212]

Hi pgregor, how did you solved it? I have the same problem