900D to FGVM Throughput over 1GB P2P

[Smartypants]

I'm trying to find the root cause to a throughput problem between two Fortigate firewalls a HA pair of 900D's at the organizations main building and a remote data center.

With the help of several qualified engineers at the remote hosting site that hosts servers for 100's of customers two connections to the data center been provisioned. The primary is a 1GB AT&T P2P and for backup is IPsec VPN over a 500/500 Internet circuit.

Both circuits test as being fine; no interface configuration errors or latency problems and according to the ISP's the performance matches spec., but with that said neither circuit is usable to connect users in San Francisco to their data in a Irvine data center.

Both the 500/500 MB/s IPsec tunnel and the 1000 MB/s are unable to even come close to that promised performance having an asymmetrical throughput of about 112 MB/s IN and only 4.5 MB/s OUT.

I won't go into to all of the changes of SFP's, patch cables etc I've done trying to fix this but I will say the reason I'm convinced that the problem is local to the 900D and not somehow the remote data center is that I have a similar throughput problem between the 900D and the Office 365 cloud in that 20GB mailbox takes about 24-hours to transfer.

Is it possible that the Fortigate firewall is to blame?

Its primarily being used as a internal firewall separating two different user VLANs from four resource VLANs for PCI-DSS compliance so I have a ton of IP4 policies and use features like FSSO and I'm using many of the UTM firewall features were useful and the VLAN "routing" is happening on the 900D's.

Memory on the 900D's is around 60%

CPU on the 900D's is always less than 15%

I'm thinking that the firewall is overloaded is that possible and how do I test it to find out?

[Toshi_Esumi]

I made the exactly same comment below at another thread about slow vpn issue:

Check through the points I commented in the thread below: https://forum.fortinet.co...p;m=151196&mpage=1

[Smartypants]

Yes helpful and I did read this soon after it was posted. Thanks!

Keep in mind that my IPsec VPN tunnel between the two Fortigate firewalls was over my ISP's 500/500 MB/s circuit but the fact that the 1GB P2P (different circuit and different ISP) was also having the same problem reduces most of the normal down to ether the Firewall or data-center.

I try not to get to hung up with the Data-center being the cause because I have the same issue with Office 365.

I opened a support ticket with Fortigate Support yesterday AM and as of today they haven't called me back but some-other engineers (not Fortigate) though that I might be overloading the 900D's.

We only have a 200 users but we do I have a lot of IP4 Policies and UTM stuff going on.

But getting only 65-112 MB/s one way and 4.5 MB/s the other over a 500/500 IPsec tunnel AND over a 1GB P2P is not good.

BYW, speedtest.net I show 480/465 speed. If I use one of the other http5 tests that use larger data-sets the throughput drops dramatically.  

[emnoc]

do I have a lot of IP4 Policies and UTM stuff going on.

what exactly?

IMHO ( probably can't be done now ), you should really benchmark these b4 they are in production and carrying traffic

iperf3/D-ITG for example  and a simple traffic flow  in/out out/in with and without a utmprofile. Than you have a baseline on the raw thruput.

next, place two FGT back-2-back and set a simple IPSEC-tunnel, again iperf3/D-ITG for  packet generation and gather a benchmark with and|or withut utm profiles

just my 2cts

FWIW we have FGT900D running over 2-3gbps during the day, no UTMprofiles enabled, no ipsec/sslvpn. We are using 10GIGE interfaces only. It's a powerful box for sure.

Ken

[Toshi_Esumi]

By the way, 900D seems to have two NP6s for encryption/decryption acceleration just like our 1500Ds. That means other firewall features wouldn't affect much to vpn throughput as long as they're handed off to an NP6. Although I don't think it's the main factor for your situation right now because the number is way lower than any performance issues (I believe in significant packet loss somewhere), I want to point out that which port you use for in and out along the path encrypted/decrypted packets go through actually affect to the vpn performance especially under multi-vdom environment. We had to change our design to use only one NP6 (out of two) not to cause any handoff between NP6_0 to/from NP6_1 involving the CPU for the same packet.

Check the documentation below for the 900D section and review your design if it's optimum.

http://docs.fortinet.com/...re-acceleration-54.pdf

[Toshi_Esumi]

When you do speedtest, pay attention to how the test packets get to/come out of the test server. We had some customers who got AT&T ASE circuits at their location and coming into our network before getting out to the internet (we are a part of their internet circuit although AT&T provides the "last mile") and had been having the speed issue for one direction. Those commonly ended up AT&T equipment having duplex mismatch at the last hop.

If you don't mind, please post the outcome of FTNT investigation. We in the past went through L1->L2->L3 escalations that took several months and found no problem on both sides of FGs.

[Smartypants]

To late to benchmark.

I do have a question about how the UTM performance levels are calculated.

The 900D has 32 1-GB Ports and 2 10-GB Ports and because we purchased the 900D to do internal segmentation to limit PCI-DSS scope.

 As I read the Fortigate 900D data sheet I see that the Firewall is rated at 52GBps but the IPS is 4.2 GBps, NGFW 4 GBps and Threat Protection is at 3 GBps so what does that mean?

Looking at the IP4 Policies and sorting them by sequence I show 454 different IP4 Policies sequences. I know that's a lot of policies.

So some policies have IPS, NGFW, etc applied others don't does it matter how many of the policies have UTM features enabled or do these ratings mean that if a policy has a UTM applied is that the total throughput for a UTM period.

If I have NGFW applied to all 454 sequences would the total throughput across all be 4GBps total or 8.8 Mbps per Policy sequence 

[emnoc]

As I read the Fortigate 900D data sheet I see that the Firewall is rated at 52GBps but the IPS is 4.2 GBps, NGFW 4 GBps and Threat Protection is at 3 GBps so what does that mean?

Using industry  benchmark and packets sizes 64-1500 you have 52gig if you ran it as a pure FW &  with no UTM. As soon as you run any of the other features,   you thru-put will drop to the level listed.

If I have NGFW applied to all 454 sequences would the total throughput across all be 4GBps total or 8.8 Mbps per Policy sequence

Not following your logic , but the thru-put is not per fw-policy  or IP, but by that the total thruput if NGFW  features are used and max at 4Gbps

BTW FTNT over rate the products at best real-world value imho.I would look for a 3rd party benchmark  report like those from  NSSlanbs or so if you want real-life numbers.

Or

You have to put together you own demo. I would not take the word of the FTNTsalesTeam without  validation. Review and Validation is what you need ;)

And finally  454 fwpolicy is nothing, I'm currently login to a FGT-FW  now &  that has  12359k policies

Ken

[Smartypants]

Thanks Ken,

You managed to answer my badly formatted questions.

and hearing that my policy count didn't cause your hair light on fire was reassuring.

I'll post an update after Fortigate support is done having me send test results

-Steve

[MikePruett]

Are you checking hardware stats to make sure all NPU/CPU/ASIC etc are fine? I had a case where a Gate was nuking itself because the two heavily used ports were on the same CPU and it was causing issues.

Please do keep us updated on this though. Would like to see the cause of your issue.