I'm trying to find the root cause to a throughput problem between two Fortigate firewalls a HA pair of 900D's at the organizations main building and a remote data center.
With the help of several qualified engineers at the remote hosting site that hosts servers for 100's of customers two connections to the data center been provisioned. The primary is a 1GB AT&T P2P and for backup is IPsec VPN over a 500/500 Internet circuit.
Both circuits test as being fine; no interface configuration errors or latency problems and according to the ISP's the performance matches spec., but with that said neither circuit is usable to connect users in San Francisco to their data in a Irvine data center.
Both the 500/500 MB/s IPsec tunnel and the 1000 MB/s are unable to even come close to that promised performance having an asymmetrical throughput of about 112 MB/s IN and only 4.5 MB/s OUT.
I won't go into to all of the changes of SFP's, patch cables etc I've done trying to fix this but I will say the reason I'm convinced that the problem is local to the 900D and not somehow the remote data center is that I have a similar throughput problem between the 900D and the Office 365 cloud in that 20GB mailbox takes about 24-hours to transfer.
Is it possible that the Fortigate firewall is to blame?
Its primarily being used as a internal firewall separating two different user VLANs from four resource VLANs for PCI-DSS compliance so I have a ton of IP4 policies and use features like FSSO and I'm using many of the UTM firewall features were useful and the VLAN "routing" is happening on the 900D's.
Memory on the 900D's is around 60%
CPU on the 900D's is always less than 15%
I'm thinking that the firewall is overloaded is that possible and how do I test it to find out?
Nominating a forum post submits a request to create a new Knowledge Article based on the forum post topic. Please ensure your nomination includes a solution within the reply.
I made the exactly same comment below at another thread about slow vpn issue:
Check through the points I commented in the thread below: https://forum.fortinet.co...p;m=151196&mpage=1
Yes helpful and I did read this soon after it was posted. Thanks!
Keep in mind that my IPsec VPN tunnel between the two Fortigate firewalls was over my ISP's 500/500 MB/s circuit but the fact that the 1GB P2P (different circuit and different ISP) was also having the same problem reduces most of the normal down to ether the Firewall or data-center.
I try not to get to hung up with the Data-center being the cause because I have the same issue with Office 365.
I opened a support ticket with Fortigate Support yesterday AM and as of today they haven't called me back but some-other engineers (not Fortigate) though that I might be overloading the 900D's.
We only have a 200 users but we do I have a lot of IP4 Policies and UTM stuff going on.
But getting only 65-112 MB/s one way and 4.5 MB/s the other over a 500/500 IPsec tunnel AND over a 1GB P2P is not good.
BYW, speedtest.net I show 480/465 speed. If I use one of the other http5 tests that use larger data-sets the throughput drops dramatically.
do I have a lot of IP4 Policies and UTM stuff going on.
what exactly?
IMHO ( probably can't be done now ), you should really benchmark these b4 they are in production and carrying traffic
iperf3/D-ITG for example and a simple traffic flow in/out out/in with and without a utmprofile. Than you have a baseline on the raw thruput.
next, place two FGT back-2-back and set a simple IPSEC-tunnel, again iperf3/D-ITG for packet generation and gather a benchmark with and|or withut utm profiles
just my 2cts
FWIW we have FGT900D running over 2-3gbps during the day, no UTMprofiles enabled, no ipsec/sslvpn. We are using 10GIGE interfaces only. It's a powerful box for sure.
Ken
PCNSE
NSE
StrongSwan
By the way, 900D seems to have two NP6s for encryption/decryption acceleration just like our 1500Ds. That means other firewall features wouldn't affect much to vpn throughput as long as they're handed off to an NP6. Although I don't think it's the main factor for your situation right now because the number is way lower than any performance issues (I believe in significant packet loss somewhere), I want to point out that which port you use for in and out along the path encrypted/decrypted packets go through actually affect to the vpn performance especially under multi-vdom environment. We had to change our design to use only one NP6 (out of two) not to cause any handoff between NP6_0 to/from NP6_1 involving the CPU for the same packet.
Check the documentation below for the 900D section and review your design if it's optimum.
When you do speedtest, pay attention to how the test packets get to/come out of the test server. We had some customers who got AT&T ASE circuits at their location and coming into our network before getting out to the internet (we are a part of their internet circuit although AT&T provides the "last mile") and had been having the speed issue for one direction. Those commonly ended up AT&T equipment having duplex mismatch at the last hop.
If you don't mind, please post the outcome of FTNT investigation. We in the past went through L1->L2->L3 escalations that took several months and found no problem on both sides of FGs.
To late to benchmark.
I do have a question about how the UTM performance levels are calculated.
The 900D has 32 1-GB Ports and 2 10-GB Ports and because we purchased the 900D to do internal segmentation to limit PCI-DSS scope.
As I read the Fortigate 900D data sheet I see that the Firewall is rated at 52GBps but the IPS is 4.2 GBps, NGFW 4 GBps and Threat Protection is at 3 GBps so what does that mean?
Looking at the IP4 Policies and sorting them by sequence I show 454 different IP4 Policies sequences. I know that's a lot of policies.
So some policies have IPS, NGFW, etc applied others don't does it matter how many of the policies have UTM features enabled or do these ratings mean that if a policy has a UTM applied is that the total throughput for a UTM period.
If I have NGFW applied to all 454 sequences would the total throughput across all be 4GBps total or 8.8 Mbps per Policy sequence
As I read the Fortigate 900D data sheet I see that the Firewall is rated at 52GBps but the IPS is 4.2 GBps, NGFW 4 GBps and Threat Protection is at 3 GBps so what does that mean?
Using industry benchmark and packets sizes 64-1500 you have 52gig if you ran it as a pure FW & with no UTM. As soon as you run any of the other features, you thru-put will drop to the level listed.
If I have NGFW applied to all 454 sequences would the total throughput across all be 4GBps total or 8.8 Mbps per Policy sequence
Not following your logic , but the thru-put is not per fw-policy or IP, but by that the total thruput if NGFW features are used and max at 4Gbps
BTW FTNT over rate the products at best real-world value imho.I would look for a 3rd party benchmark report like those from NSSlanbs or so if you want real-life numbers.
Or
You have to put together you own demo. I would not take the word of the FTNTsalesTeam without validation. Review and Validation is what you need ;)
And finally 454 fwpolicy is nothing, I'm currently login to a FGT-FW now & that has 12359k policies
Ken
PCNSE
NSE
StrongSwan
Thanks Ken,
You managed to answer my badly formatted questions.
and hearing that my policy count didn't cause your hair light on fire was reassuring.
I'll post an update after Fortigate support is done having me send test results
-Steve
Are you checking hardware stats to make sure all NPU/CPU/ASIC etc are fine? I had a case where a Gate was nuking itself because the two heavily used ports were on the same CPU and it was causing issues.
Please do keep us updated on this though. Would like to see the cause of your issue.
Mike Pruett
Select Forum Responses to become Knowledge Articles!
Select the “Nominate to Knowledge Base” button to recommend a forum post to become a knowledge article.
User | Count |
---|---|
1633 | |
1063 | |
751 | |
443 | |
210 |
The Fortinet Security Fabric brings together the concepts of convergence and consolidation to provide comprehensive cybersecurity protection for all users, devices, and applications and across all network edges.
Copyright 2024 Fortinet, Inc. All Rights Reserved.