Support Forum
The Forums are a place to find answers on a range of Fortinet products from peers and product experts.
TheJaeene
Contributor

sFlow breaks IPsec Traffic (Offloading) on a 60D !?!

Hello Folks,

 

today I configured sFlow on a 60D, afterwards I noticed high ESP-Errors (Replayed-Packets) on a far-end VPN Tunnel Fortigate.

For me it seems that enabling sFlow on this small SoC Boxes breaks the IPSec "Acceleration".

 

The Box was running 5.2.2 and the sFlow Polling-Interval was configured quite conservative.

CPU-Load was O.K.

 

Has anyone seen similar issues yet?

 

 

Regards,

 

Jan

1 Solution
Christopher_McMullan

I'll weigh in here. Neat to research the issue, so thank you for the opportunity!

 

The problem is an existing offloading situation when sFlow is applied. What initially seems counter-intuitive is that the issue is not sFlow-specific. Traffic *was* being offloaded historically, but when enabling sFlow, you are essentially queuing new traffic to be forced through the kernel instead (i.e., the CPU).

 

A reboot fixes the issue because, when the tunnels come back up, they are all functioning through the CPU - no more offloading. You may have noticed higher baseline CPU usage, depending on your traffic patterns.

 

A bug is open, but it could always use more customer reporting and buy-in for a solution. If you end up opening a ticket with TAC, quote the number here so that I can make an internal note referencing the bug.

Regards, Chris McMullan Fortinet Ottawa

View solution in original post

9 REPLIES 9
emnoc
Esteemed Contributor III

Open a case with  TAC. I'm curious if you use netflow  does the same thing happen?

 

And are you 100% sure the ESP Replayed-Packets is not cause by something else?

 

And is the far-end device a cisco ?

 

If your bored you could probably do a small packet capture of the ESP  traffic and check the sequences.

PCNSE 

NSE 

StrongSwan  

PCNSE NSE StrongSwan
TheJaeene
Contributor

Hi emnoc,

 

 

once I activated sflow sampling on the Interface (WAN, not Tunnel IF) the ESP-Errors were visible on Remote Fortigates (80C, 100D). All VPN Traffic going through that Interface was very sluggish.  (Several Tunnels)

 

Disabling the sflow sampling didn´t help, only a reboot cured the behaviour.

 

And yes, I could reproduce it several times

 

 

Once our FGT-Support account is repaired, I´ll open a Ticket.

 

Regards,

Jan

 

 

emnoc
Esteemed Contributor III

So  you have replay enabled on your vpn-tunnel ph2 cfgs? This could be a bug or something else going on. TAC should look into it and see what's going on.

 

What happens if you enable sFlow on the bigger fortigates, do you see errors on the opposite end ?  Might be hardware related on the smaller units.

PCNSE 

NSE 

StrongSwan  

PCNSE NSE StrongSwan
TheJaeene
Contributor

I also assume that this is a SoC FGT related Problem.

Replay-Detection ist enabled in the Ph2 Configs.

 

Right now our bigger Lab Units are not available for testing.

 

But during Research I found some old npu/nplite related stuff that I now disabled.

 

    set enc-offload-antireplay enable *gone now*     set offload-ipsec-host enable *gone now*

 

 

I´ll check again if the problem still persits without offloading antireplay to the nplite.

Christopher_McMullan

I'll weigh in here. Neat to research the issue, so thank you for the opportunity!

 

The problem is an existing offloading situation when sFlow is applied. What initially seems counter-intuitive is that the issue is not sFlow-specific. Traffic *was* being offloaded historically, but when enabling sFlow, you are essentially queuing new traffic to be forced through the kernel instead (i.e., the CPU).

 

A reboot fixes the issue because, when the tunnels come back up, they are all functioning through the CPU - no more offloading. You may have noticed higher baseline CPU usage, depending on your traffic patterns.

 

A bug is open, but it could always use more customer reporting and buy-in for a solution. If you end up opening a ticket with TAC, quote the number here so that I can make an internal note referencing the bug.

Regards, Chris McMullan Fortinet Ottawa

kelv1n
New Contributor

OMG THANK YOU!!!

 

I've been going insane for the past several days, trying to figure why our 2 sites connected via a 1Gbp/s link would carry 600Mb/s in 1 direction, but only 50-150Mb/s in the other (it is quite amazing how much performance the NPU seems to deliver).. turns out traffic was not offloading to the NPU, and removing our Netflow and Sflow configs fixed this.

Delta
New Contributor

I have a 90D ... just enabled sflow and ... tunnels stopped passing traffic.  Is there a workaround yet ?  (5.3)

Thought for the day: Advertising (n): the science of arresting the human intelligence for long enough to get money from it. -- Stephen Leacock.
Thought for the day: Advertising (n): the science of arresting the human intelligence for long enough to get money from it. -- Stephen Leacock.
Christopher_McMullan

Once it's done, it's done, unfortunately. The workaround is to reboot the FortiGate at this point.

 

The bug is open, but not fixed yet. I'd encourage you to open a TAC case to put your weight behind a quicker schedule for the fix. Not to say there is a schedule, but something about squeaky wheels comes to mind.

Regards, Chris McMullan Fortinet Ottawa

Delta
New Contributor

Do you have a bug # to reference for TAC

Thought for the day: Advertising (n): the science of arresting the human intelligence for long enough to get money from it. -- Stephen Leacock.
Thought for the day: Advertising (n): the science of arresting the human intelligence for long enough to get money from it. -- Stephen Leacock.
Top Kudoed Authors