VPN thinks its dead when its not. Causing WAN traffic to suddenly stop.

[sysneeb]

Been having problem shifting over to our new WAN environment consisting of Fortigate Routers. The VPN that is connecting the branch seems to just suddenly drop dead when the underlying network is alive).

For clarification below is our current environment that were trying to shift over to.

1. 3 branches (ill just call them A,B,C) with Branch A being the main office
        B<>A<>C (B and C only has VoIP traffic going toward each other)

1. 2 Fortigate Router in each branch, these routers are in Active/Standby mode
2. The Traffic that’s flowing is numerous including HTTPS, VoIP, RDP etc.
3. Fortigate we are using is 60F
4. We have had to upgrade our FortiOS numerous times.
5. We have reverted back to our old setup about 3 times per branch

First incident.

We were using FortiOS 6.4.8 for all branches, with this, we were using Policy Based Routing as well as SDWAN. When the initial turn over to the new fortigate was executed to all 3 branches, it had seemed to be working fine. That was until that night when the traffic just stopped flowing and the VPN seemed to just die. But the underlying internet connection (2 x 1Gbps) was alive. After contacting support, we were told there was a bug with PBR on the version we were using (6.4.8) So we updated it 7.0

Second Incident.

After giving it some time to update our OS to 7.0, we decided not to use PBR for the second turn over plan and just use SDWAN with normal routing configured on the routers. The second turn over seemed to execute fine just like the first time but after some time it died again like the first time. Now, for this turn over, we only shifted the environment from Branch A to Branch B, Branch C was still using the old WAN set up.

Third Incident

This time the OS was still 7.0 but we stopped using SDWAN. Seemed fine after shifting to all Fortigate environment but after sometime it died. Now, for this turn over, we only shifted the environment from Branch A to Branch B, Branch C was still using the old WAN set up.

Fourth Incident

We suspected the 7.0 had either a bug or it was a faulty lot. Since we can’t figure out if it is a faulty lot, we decided to revert back to version 6.4.8 and this time we did not use PBR or SDWAN. After executing the shift, it seemed to work fine but after sometime (usually a few hours to a few days) the VPN dies (underlying internet is alive) and the traffic stopped flowing. Now, for this turn over, we only shifted the environment from Branch A to Branch B, Branch C was still using the old WAN set up.

Fifth Incident(kind of)

This time we updated the FortiOS to 6.4.9 on the branch B and A, and this time we decided not to send a certain traffic between A and B, this traffic consist of video data. After making the change to Forti Setup, it seemed to work fine. After 2 weeks its still working fine so we figured it’s the traffic that’s causing the down state but we haven’t looked into it too much yet.

Sixth Incident

Since we were abel to make the change to Forti Setup between branch A and B we figured A and C would work fine with the same concept. We made the shift to the Forti setup between A and C just TODAY. After the turn over it seemed to work for a few hours but just about an hour ago, the VPN died again (again, the underlying internet connection was alive) .

We seriously have no idea what the hell is happening, we went as far as not using fancy features and just using Fortigate as it was intended, which was routing with a bit of VPN for WAN usage.

If anyone can give us an feedback on what could be happening. I can give you guys more information if needed. If this keeps persisting, we cannot make outrchange to all FortiOS routers between branches.

Thanks in advance.

edit:

when i say the VPN dies, i mean the VPN connection seems to be alive but the there is no traffic going through what so ever after sometime and we dont know what triggers it

[Anonymous]

Hello @sysneeb ,

Thank you for posting on the Fortinet Forums. As per your query, we might need to see how the SDWAN health check parameters/SLAs are set. 

Also during the time of issue were you able to get the session list from FortiGate?

Thanks

[sysneeb]

Current we are not using SDWAN and just connecting branches with a simple VPN connection.

Now having said that, were not closely looking at the session list per se, but at the busiest its around 4000 sessions which is not enough to kill a vpn or make the vpn go hay wire, in my personal opinion.

When the traffic suddenyl stops, the traffic from the main branch to other branch seems to be passing but the return packet to the main branch seems to drop, or the main branch isnt receiving any. at least thats what the support says, but the response time is a little slow hence i am also asking here.

Also i looked up on reddit and came across this post, do you think this might resolve it? 

Site to Site VPN woes with 60F : fortinet (reddit.com)

Also i saw there is a known issue in 6.4.9 where the ESP packet doesnt receive on one end when the offloaded session isnt deleted, can this have something to do with what we are experiecing?

regards.

[boneyard]

it is quite difficult to guess what the issue is. it could be many things.

turning off npu-offload is a trouble shooting step often tried, so you can give that a go for sure.

it doesnt feel normal, it shouldn't happen. but trouble shooting requires full information and a forum isnt the most logical place for that. it is probably better to involve fortinet support for this.

[Anonymous]

Hello @sysneeb ,

The link you provided above can be a solution to this and it is better explained in the links below.

https://community.fortinet.com/t5/FortiGate/Ensuring-IPSec-traffic-is-offloaded-for-improved-through...

https://docs.fortinet.com/document/fortigate/6.4.8/hardware-acceleration/636026/disabling-np-offload...

However, I would be able to give my expertise if I would be able to see the session list at the time of issue or debugs of the packet flow for this traffic.

The following link shows how to run the debugs for the packet flow.

https://community.fortinet.com/t5/FortiGate/Troubleshooting-Tip-First-steps-to-troubleshoot-connecti...

[Muhammad_Haiqal]

Hi there,

I can see you mentioned about SDWAN, Policy Route, VPN.
Based on the behaviour, im suspecting this issue is due to asymmetric routing.

There are many point need to check here.
A, B and C.

This KB might get you some idea on the troubleshooting.

https://kb.fortinet.com/kb/microsites/microsite.do?cmd=displayKC&docType=kc&externalId=FD50568

[sysneeb]

in the early stage of the shift we were using all 3 feature you mentioned and the asymmetrical routing was a problem, we ended up upping the version 7.0.2, but now we are onyl using a simple VPN which results in the same kind of behaviour (VPN up but not traffic)

[Muhammad_Haiqal]

Hi Sysneeb,

Its the best to get a proper support for this. Need to verify if the issue is on this fortigate or not.

In some cases, the issue is not on the Fortigate, but switch send out traffic to another gateway. So traffic basically not reaching the Fortigate.

I would suggest to call fortinet support hotline here:

https://www.fortinet.com/support/contact.html

[sysneeb]

thank you all for the comments

upon doing bit of research until it happens next time i cannot get the session list or the log.

what do the following 2 command do? will it help with the log caputring? also am i able to keep either of em running at all times?

diagnose sniffer packet any "proto 50" 4 0 a

diagnose debug application ike -1