sFlow breaks IPsec Traffic (Offloading) on a 60D !?!

[TheJaeene]

Hello Folks,

today I configured sFlow on a 60D, afterwards I noticed high ESP-Errors (Replayed-Packets) on a far-end VPN Tunnel Fortigate.

For me it seems that enabling sFlow on this small SoC Boxes breaks the IPSec "Acceleration".

The Box was running 5.2.2 and the sFlow Polling-Interval was configured quite conservative.

CPU-Load was O.K.

Has anyone seen similar issues yet?

Regards,

Jan

[emnoc]

Open a case with  TAC. I'm curious if you use netflow  does the same thing happen?

And are you 100% sure the ESP Replayed-Packets is not cause by something else?

And is the far-end device a cisco ?

If your bored you could probably do a small packet capture of the ESP  traffic and check the sequences.

[TheJaeene]

Hi emnoc,

once I activated sflow sampling on the Interface (WAN, not Tunnel IF) the ESP-Errors were visible on Remote Fortigates (80C, 100D). All VPN Traffic going through that Interface was very sluggish.  (Several Tunnels)

Disabling the sflow sampling didn´t help, only a reboot cured the behaviour.

And yes, I could reproduce it several times

Once our FGT-Support account is repaired, I´ll open a Ticket.

Regards,

Jan

[emnoc]

So  you have replay enabled on your vpn-tunnel ph2 cfgs? This could be a bug or something else going on. TAC should look into it and see what's going on.

What happens if you enable sFlow on the bigger fortigates, do you see errors on the opposite end ?  Might be hardware related on the smaller units.

[TheJaeene]

I also assume that this is a SoC FGT related Problem.

Replay-Detection ist enabled in the Ph2 Configs.

Right now our bigger Lab Units are not available for testing.

But during Research I found some old npu/nplite related stuff that I now disabled.

    set enc-offload-antireplay enable *gone now*     set offload-ipsec-host enable *gone now*

I´ll check again if the problem still persits without offloading antireplay to the nplite.

[Christopher_McMullan · Accepted solution]

I'll weigh in here. Neat to research the issue, so thank you for the opportunity!

The problem is an existing offloading situation when sFlow is applied. What initially seems counter-intuitive is that the issue is not sFlow-specific. Traffic *was* being offloaded historically, but when enabling sFlow, you are essentially queuing new traffic to be forced through the kernel instead (i.e., the CPU).

A reboot fixes the issue because, when the tunnels come back up, they are all functioning through the CPU - no more offloading. You may have noticed higher baseline CPU usage, depending on your traffic patterns.

A bug is open, but it could always use more customer reporting and buy-in for a solution. If you end up opening a ticket with TAC, quote the number here so that I can make an internal note referencing the bug.

[kelv1n]

OMG THANK YOU!!!

I've been going insane for the past several days, trying to figure why our 2 sites connected via a 1Gbp/s link would carry 600Mb/s in 1 direction, but only 50-150Mb/s in the other (it is quite amazing how much performance the NPU seems to deliver).. turns out traffic was not offloading to the NPU, and removing our Netflow and Sflow configs fixed this.

[Delta]

I have a 90D ... just enabled sflow and ... tunnels stopped passing traffic.  Is there a workaround yet ?  (5.3)

[Christopher_McMullan]

Once it's done, it's done, unfortunately. The workaround is to reboot the FortiGate at this point.

The bug is open, but not fixed yet. I'd encourage you to open a TAC case to put your weight behind a quicker schedule for the fix. Not to say there is a schedule, but something about squeaky wheels comes to mind.

[Delta]

Do you have a bug # to reference for TAC