Slow site to site VPN Performance

[mkintexas]

Folks,

Recently my company decided to save money by transitioning away from MPLS and metro ethernet based connectivity to Internet based site to site VPN's.    For our stores we are installing Time Warner and Comcast business class Internet.  Generally either 100/10 or 100/20 with one location being on a Comcast fiber based Internet circuit that is 30/30.

So far our experience has not been all that great.  Our data center currently has a 100/100 fiber based Internet connection (1g to be installed next week).  Our 100mb is not oversubscribed at this point.  Whenever I try to do a windows based drag and drop from the data center to the store on average I get from 1.5-2.5MB on the copy.  so basically 12-20 megabit despite the fact that my store has a 100mb download pipe.  If I try to use FTP over the VPN I get the same speed.  However if I take the same server at the DC and do a 1 to 1 NAT and then FTP to it from the same store over the Internet and not through the VPN I see close to the 100mb speed that we are subscribed to.  Interstingly whenever I copy from the store to the DC I almost always get the full 20mb upload speed.   Finally at our 30/30 store I get all 30mb both directions.  

My data center has a 500D and all of my stores have a 140D.  So I would think there is enough horsepower to be able to handle the occasionally large file copy.  We don't generally move a lot of data over our VPN's.  Mainly web based applications with some videos.  However when we need it, it would be nice to have a nice file copy speed.  I understand there is some overhead on VPN's but not to this degree.   I have already tried various MTU sizes on WAN interfaces at both the DC and my lab store.

At this point I am stumped.  Why is my VPN running so slowly?  Is it possible that TW and/or comcast throttles UDP 500/4500 or the ESP protocol?   At this point I along with our CIO is ready to abort this project and go with Fiber in all 90 locations.  But I am not quite ready to give up.  

Any help would be appreciated.

Mark

[nothingel]

emnoc wrote:

What do you  mean by tricking. ESP proto#50 is what the ipsec-sa are using ( aka  phase2 ), phase1 ( ike or Ikasmp ) is where NAT-T and udp comes into play. You can't trick anything as far as this goes. When the twp vpn-peers 1st engage in ( phase1 ) it learns if NAT-T is involved and switch  from  udp/500---udp/500 to just udp/4500

I think we might be saying the same thing.  My entire point is that I needed to ensure the tunnels (phase2) are wrapped inside udp/4500 instead of ESP protocol 50.  Comcast cable to Comcast fiber performance issues are resolved when udp/4500 is used in phase2.  When going cable-to-cable or fiber-to-fiber, ESP protocol 50 performs as expected.

[mkintexas]

I am pretty sure that I have tried nat-t forced on fgt's wth no change. Would you mind send a screen shot on your modem set up I think there is something to what you did in the modem specifically I am thinking these Comcast modems can't handle ESP too well One side of mine is particularly bad my FGT log shows my wan1 interface bounce several times a day Yet that same interface does not bounce when no traffic is applied to it because i migrated to a metro E tied to wan2. So despite my 5.4 I would really like to try the port forwarding you did in your Comcast modem

[nothingel]

mkintexas wrote:
I am pretty sure that I have tried nat-t forced on fgt's wth no change. Would you mind send a screen shot on your modem set up I think there is something to what you did in the modem specifically I am thinking these Comcast modems can't handle ESP too well One side of mine is particularly bad my FGT log shows my wan1 interface bounce several times a day Yet that same interface does not bounce when no traffic is applied to it because i migrated to a metro E tied to wan2. So despite my 5.4 I would really like to try the port forwarding you did in your Comcast modem

10.1.10.2 is assigned as a secondary IP address on the wan interface attached to the cable modem.  This address is also specified as the tunnel's phase1 "local-gw" parameter.  Lastly, the remote/far-side FGT references the cable modem's public IP for its "remote-gw" parameter instead of the FGT's own IP.  I also removed the "nattraversal disable" option.  No changes are made to the phase2 on either side.

FWIW, I have considered the possibiliy that the SMC comcast modems don't perform well with ESP but yet I see the proper 20-22mbps outgoing speed via the tunnel when it's cable-to-cable using ESP protocol 50.  When switching one side to fiber (cable-to-fiber), the same test suddenly drops to 8-10mbps (also ESP protocol 50).  This seems to suggest that cable modem isn't at fault.

[mkintexas]

  Ahh.  ok.  One caveat I didn't include.   All of our coax circuits have a /29.  The last of the six of course is the default gateway we assign in our 140d.  The first available of the /29 is assigned to wan1 and then the second usable is preserved for our guests wifi. 

Am I correct in assuming that you have a single IP and that your wan interface on your fgt is assigned a non-routable and then you nat with your cable modem?  or do you have a single public ip from Comcast that you assign to your fgt?

[nothingel]

mkintexas wrote:

Am I correct in assuming that you have a single IP and that your wan interface on your fgt is assigned a non-routable and then you nat with your cable modem?  or do you have a single public ip from Comcast that you assign to your fgt?

We have a /28 with comcast.  One IP is normally assigned to the FGT's wan interface and tunnels normally refer to this IP.  The cable modem has its own unique public IP.  I also assigned 10.1.10.2 as a secondary address to the FGT's wan interface and used this address in the FGT's tunnel "local-gw" parameter. Before beginning these experiments, we were not using any of the non-routable 10.x.x.x IPs provided by the cable modem.  I hope this is helpful.

[mkintexas]

Let me make sure I understand.   you have a /28 assigned by Comcast for your coax connection.  One address for the modem of course.  Do you then have a public ip assigned to your wan interface plugged into the cable modem and then also assign a secondary IP in the 10.1.10 to that same WAN interface?

[mkintexas]

nothingel wrote:

mkintexas wrote:

Am I correct in assuming that you have a single IP and that your wan interface on your fgt is assigned a non-routable and then you nat with your cable modem?  or do you have a single public ip from Comcast that you assign to your fgt?

We have a /28 with comcast.  One IP is normally assigned to the FGT's wan interface and tunnels normally refer to this IP.  The cable modem has its own unique public IP.  I also assigned 10.1.10.2 as a secondary address to the FGT's wan interface and used this address in the FGT's tunnel "local-gw" parameter. Before beginning these experiments, we were not using any of the non-routable 10.x.x.x IPs provided by the cable modem.  I hope this is helpful.

Ok.  I attempted to replicate this port forwarding idea to see if I could get better performance.  Unfortunately I didn't see any improved performance.  So on your fgt using your coax service do you have the main WAN IP as one of the public IP's in the /28 space assigned to it and then a secondary address of 10.1.10.2.   Then the port forwarding for UDP 500 and UDP 4500.   Then you put 10.1.10.2 in the local ID parameter of your fortigate vpn phase 1 config.  

Now on your other fgt that is serviced by Comcast fiber.  Do you point it to the public ip address of the cable modem and not the public ip you assigned to the wan interface of your coax serviced fgt.  And that public IP is one of the address in the /28 space which is assigned to the modem (and your fgt uses as default gateway)?

Finally are there any other changes that you made to the comcast modem?  Did you do any 1-1 nat'ing?

Thanks,

[nothingel]

mkintexas wrote:

Finally are there any other changes that you made to the comcast modem?  Did you do any 1-1 nat'ing?

You seem to understand my setup.  And no, I didn't add any 1-to-1 NAT in the cable modem.  The cable modem config is very vanilla.  The only other thing that's not standard is disabling the "Gateway Smart Packet Detection" feature (or maybe that's now a default) but this part was disabled a long time ago therefore I don't know if it has any bearing on this issue.

Have you confirmed that your tunnel traffic is indeed using UDP now instead of ESP?  If it is then it would seem that your situation is different than mine.  My observations are based on connectivity between three sites but I have at least one more to test.  If I find anything else of interest, I'll definitely update this thread.

[mkintexas]

I am afraid I don't know exactly how to see if ESP is being wrapped in UDP packets. Is there a diag command or something that will show that?

I would think our issues are the same.  Certainly sound the same.   I have 16 business class 100/20 and 75/15 circuits with Comcast and another 17 100/10 with Time Warner Cable.  All exhibit the same behavior.  But I do have the one remote site on the 30/30 fiber as it was the only option.  Like you fiber to fiber is fine.  It is just fiber to coax that is giving me grief.  I am going to try to port forwarding at another site.   Just to clarify.  All of my sites have as /29.   One site is 50.x.y.128/29    I have assigned my wan 1 interface of the 140D .129.   The default gateway is the comcast modem at .134   I also gave that wan1 interface a secondary IP address of 10.1.10.2 (verified that the cable modem's lan ip is indeed in the 10.1.10 space).   In phase 1 I put 10.1.10.2 as the local-gw.   On my remote side do I tell it that its vpn peer is the .129 assigned to WAN1 or the .134 that is assigned to the cable modem/router?

[nothingel]

mkintexas wrote:

I am afraid I don't know exactly how to see if ESP is being wrapped in UDP packets. Is there a diag command or something that will show that?

diagnose sniffer packet wan1 'host 1.1.1.1'

Replace wan1 with your actual wan interface.  Replace the IP address with the remote FGT's IP address.  This will show ALL traffic relating to this IP and nothing else.  You can also be more specific by adding "host 1.1.1.1 and port 4500" or "host 1.1.1.1 and proto 50".  You can also use "not" to exclude as well.  Press CTRL+C to stop.  I recommend using SSH rather than the GUI's built-in console.

mkintexas wrote:

I would think our issues are the same.  Certainly sound the same.   I have 16 business class 100/20 and 75/15 circuits with Comcast and another 17 100/10 with Time Warner Cable.  All exhibit the same behavior.  But I do have the one remote site on the 30/30 fiber as it was the only option.  Like you fiber to fiber is fine.  It is just fiber to coax that is giving me grief.  I am going to try to port forwarding at another site.   Just to clarify.  All of my sites have as /29.   One site is 50.x.y.128/29    I have assigned my wan 1 interface of the 140D .129.   The default gateway is the comcast modem at .134   I also gave that wan1 interface a secondary IP address of 10.1.10.2 (verified that the cable modem's lan ip is indeed in the 10.1.10 space).   In phase 1 I put 10.1.10.2 as the local-gw.   On my remote side do I tell it that its vpn peer is the .129 assigned to WAN1 or the .134 that is assigned to the cable modem/router?

On the remote side, specify your cable modem's IP (the .134 address).