Slow site to site VPN Performance

[mkintexas]

Folks,

Recently my company decided to save money by transitioning away from MPLS and metro ethernet based connectivity to Internet based site to site VPN's.    For our stores we are installing Time Warner and Comcast business class Internet.  Generally either 100/10 or 100/20 with one location being on a Comcast fiber based Internet circuit that is 30/30.

So far our experience has not been all that great.  Our data center currently has a 100/100 fiber based Internet connection (1g to be installed next week).  Our 100mb is not oversubscribed at this point.  Whenever I try to do a windows based drag and drop from the data center to the store on average I get from 1.5-2.5MB on the copy.  so basically 12-20 megabit despite the fact that my store has a 100mb download pipe.  If I try to use FTP over the VPN I get the same speed.  However if I take the same server at the DC and do a 1 to 1 NAT and then FTP to it from the same store over the Internet and not through the VPN I see close to the 100mb speed that we are subscribed to.  Interstingly whenever I copy from the store to the DC I almost always get the full 20mb upload speed.   Finally at our 30/30 store I get all 30mb both directions.  

My data center has a 500D and all of my stores have a 140D.  So I would think there is enough horsepower to be able to handle the occasionally large file copy.  We don't generally move a lot of data over our VPN's.  Mainly web based applications with some videos.  However when we need it, it would be nice to have a nice file copy speed.  I understand there is some overhead on VPN's but not to this degree.   I have already tried various MTU sizes on WAN interfaces at both the DC and my lab store.

At this point I am stumped.  Why is my VPN running so slowly?  Is it possible that TW and/or comcast throttles UDP 500/4500 or the ESP protocol?   At this point I along with our CIO is ready to abort this project and go with Fiber in all 90 locations.  But I am not quite ready to give up.  

Any help would be appreciated.

Mark

[ede_pfau]

hi,

and welcome to the forums.

This situation is strange. The 100D feature a CP8 content processor but lack a network processor ASIC (NP6). Nevertheless, even CPU based they should sustain 100 Mbps IPsec.

Please supply the following info to clarify:

- which firmware versions are you running (HQ/branch)?

- which IPsec encryption is used in phase1, phase2?

- do the 140Ds spike in CPU load on large file transfers?

- what do you get with

diag vpn ipsec status

, HQ and branch?

What I am aiming at is that certain encryption ciphers are not hardware accelerated, due to lack of NP6 or due to implementation after the ASIC was designed. On the safe side, AES128 and SHA1 are both done in hardware and more performant than, say, 3DES and AES256.

One more (faint) possibility would be frequent renegotiations due to parameter mismatch - but you would have noticed that in the logs instantly.

Lastly, have you checked with the ISP technical support that your HQ upstream is not throttled for specific protocols? They should be able to tell you with certainty.

[mkintexas]

Thank you for responding.  Responses to your questions:

Both HQ and branch are running 5.40.   I have also tested with our DR site which is a 500d running 5.4.1.   For test purposes I tested to my lab site as well (fed by comcast biz class) which is a 140d running 5.4.1.

All my VPNS are using SHA 256 and AES 256.  Both phases

Just tested and my CPU on the branch site moved from 24% to about 31%.

Below are the images from diag vpn ipsec status.  First is branch  second is HQ.  On HQ everything below software was 0's.

I have not checked with the ISP.  Mostly because the same ISP provides the 30/30 at the site where I get all 30 from hq to site. Additionally I see the same slow behavior from my DR site that has a 1gb circuit from Alpheus rather than Comcast.  So I would doubt both would do that.  But can check with them.

I will set up a tunnel with the cipher suites above and see what I get.

Thanks,

Mark

HQ:  <Truncated to show only lines with entries under each processing type>

NP6_0 aes: 8623099264 168079844352 sha256: 8623099264 168079844352 NPU HARDWARE aes: 777924503 0 sha256: 777925152 0 aes: 195345 1306 sha256: 195343 1304 SOFTWARE:

Site: <Again, deleted all rows that had only 0's>

CP8_0 aes: 124431068 241293474 sha256: 124431068 241293474 SOFTWARE:

[mkintexas]

Ok set up a tunnel with AES128 and SHA1.  Not much change.

[emnoc]

FWIW

 FTNT  numbers for vpn ipsec are not realistic or real-world. 

[mkintexas]

Sorry.  What is FTNT?

[ede_pfau]

that's short for "Fortinet". We've got to repeat that so often...I think it's the NASDAQ caller sign as well.

"FGT" is (my) alias for "Fortigate".

I'm sorry all the obvious solutions proved wrong but it's an information gain anyway.

30% CPU seems a bit high but we don't know what else is active (UTM etc.)

What still is left for experimentation is downgrading to a 'mainstream' version like v5.2.3. Unfortunately both sides run v5.4 which still is in it's infancy (nicely put). Haven't heard exactly from IPsec problems in v5.4 but it would be worth a try. I know this needs some effort and better not be tested in production.

@emnoc: I had to smile when reading that. I agree in principle (marketing always wins over engineering) but really, 30 Mbps IPsec is a no-brainer for a FGT of any size. And a hardware throttle would work in both directions, the problem at hand looks asymmetrical.

[mkintexas]

Ah yes. FGT is my short fortigate. Appreciate the ideas. We do use UTM at our stores. In fact our policy is default deny and open websites and web apps required for business functions. A typical store will have 7 zones (wan zone has two IPSec interfaces and a gre interface) the rest are just vlan interfaces. Each store only has 2 or 3 web filter profiles and 2 or 3 app filter profiles, an a/v profile, a deep inspection profile and an IPS profile. We have in the neighborhood of 150 policies on each 140d. 2 of the zones are allowed Internet access and of course have a web/app/AV/ssl inspection profile. Nothing too crazy for a 140d I wouldn't think. So that makes me wonder if companies like TWC and Comcast see IPSec traffic and throttle it or mark it best effort to make it discard eligible above all other traffic in a congested network. So another question is there any way to configure Fortinet site to site VPN's to use another port. I vaguely recall in Cisco world you could do an IPSec tunnel over tcp/10000. That may have been client based. The idea would be to trick the provider into thinking it was something else other than a VPN.

[mkintexas]

Didn't see all of your message. Downgrading would be challenging as my 500's are all production. I will say that I did connect my 140d and 500d together via a common switch and built a tunnel taking the Internet out of the picture. I don't recall the exact throughput but it certainly exceeded 100mb for sure. So the units can do it.

[Nils]

Are you able to see if your packets are fragmented? 

Sounds like there is something when you add the extra IPSEC overhead.