Skip to main content
mkintexas
mkintexas
New Member
September 24, 2016
Question

Slow site to site VPN Performance

  • September 24, 2016
  • 2 replies
  • 67005 views

Folks,

 

Recently my company decided to save money by transitioning away from MPLS and metro ethernet based connectivity to Internet based site to site VPN's.    For our stores we are installing Time Warner and Comcast business class Internet.  Generally either 100/10 or 100/20 with one location being on a Comcast fiber based Internet circuit that is 30/30.

 

So far our experience has not been all that great.  Our data center currently has a 100/100 fiber based Internet connection (1g to be installed next week).  Our 100mb is not oversubscribed at this point.  Whenever I try to do a windows based drag and drop from the data center to the store on average I get from 1.5-2.5MB on the copy.  so basically 12-20 megabit despite the fact that my store has a 100mb download pipe.  If I try to use FTP over the VPN I get the same speed.  However if I take the same server at the DC and do a 1 to 1 NAT and then FTP to it from the same store over the Internet and not through the VPN I see close to the 100mb speed that we are subscribed to.  Interstingly whenever I copy from the store to the DC I almost always get the full 20mb upload speed.   Finally at our 30/30 store I get all 30mb both directions.  

 

My data center has a 500D and all of my stores have a 140D.  So I would think there is enough horsepower to be able to handle the occasionally large file copy.  We don't generally move a lot of data over our VPN's.  Mainly web based applications with some videos.  However when we need it, it would be nice to have a nice file copy speed.  I understand there is some overhead on VPN's but not to this degree.   I have already tried various MTU sizes on WAN interfaces at both the DC and my lab store.

 

At this point I am stumped.  Why is my VPN running so slowly?  Is it possible that TW and/or comcast throttles UDP 500/4500 or the ESP protocol?   At this point I along with our CIO is ready to abort this project and go with Fiber in all 90 locations.  But I am not quite ready to give up.  

 

Any help would be appreciated.

 

Mark

    2 replies

    ede_pfau
    SuperUser
    ede_pfau
    SuperUser
    September 25, 2016

    hi,

    and welcome to the forums.

     

    This situation is strange. The 100D feature a CP8 content processor but lack a network processor ASIC (NP6). Nevertheless, even CPU based they should sustain 100 Mbps IPsec.

     

    Please supply the following info to clarify:

    - which firmware versions are you running (HQ/branch)?

    - which IPsec encryption is used in phase1, phase2?

    - do the 140Ds spike in CPU load on large file transfers?

    - what do you get with 

    diag vpn ipsec status
    , HQ and branch?

     

    What I am aiming at is that certain encryption ciphers are not hardware accelerated, due to lack of NP6 or due to implementation after the ASIC was designed. On the safe side, AES128 and SHA1 are both done in hardware and more performant than, say, 3DES and AES256.

     

    One more (faint) possibility would be frequent renegotiations due to parameter mismatch - but you would have noticed that in the logs instantly.

    Lastly, have you checked with the ISP technical support that your HQ upstream is not throttled for specific protocols? They should be able to tell you with certainty.

    mkintexas
    mkintexasAuthor
    New Member
    September 25, 2016

    Thank you for responding.  Responses to your questions:

    Both HQ and branch are running 5.40.   I have also tested with our DR site which is a 500d running 5.4.1.   For test purposes I tested to my lab site as well (fed by comcast biz class) which is a 140d running 5.4.1.

    All my VPNS are using SHA 256 and AES 256.  Both phases

    Just tested and my CPU on the branch site moved from 24% to about 31%.

    Below are the images from diag vpn ipsec status.  First is branch  second is HQ.  On HQ everything below software was 0's.

     

    I have not checked with the ISP.  Mostly because the same ISP provides the 30/30 at the site where I get all 30 from hq to site. Additionally I see the same slow behavior from my DR site that has a 1gb circuit from Alpheus rather than Comcast.  So I would doubt both would do that.  But can check with them.

     

    I will set up a tunnel with the cipher suites above and see what I get.

     

    Thanks,

    Mark

     

    HQ:  <Truncated to show only lines with entries under each processing type>

    NP6_0 aes: 8623099264 168079844352 sha256: 8623099264 168079844352 NPU HARDWARE aes: 777924503 0 sha256: 777925152 0 aes: 195345 1306 sha256: 195343 1304 SOFTWARE:

     

    Site: <Again, deleted all rows that had only 0's>

    CP8_0 aes: 124431068 241293474 sha256: 124431068 241293474 SOFTWARE:

    mkintexas
    mkintexasAuthor
    New Member
    September 25, 2016

    Ok set up a tunnel with AES128 and SHA1.  Not much change.

    emnoc
    emnoc
    New Member
    October 3, 2016

    I am afraid I don't know exactly how to see if ESP is being wrapped in UDP packets. Is there a diag command or something that will show that?

     

    ESP is it's own protocol , like tcp,udp,gre,icmp  ....and is not encapsulated into another protocol. IKSAKMP uses UDP.

     

    wiki or google ESP and protocol #50  or protocol #51 ( AH )

     

    mkintexas
    mkintexasAuthor
    New Member
    October 3, 2016

    emnoc wrote:

    I am afraid I don't know exactly how to see if ESP is being wrapped in UDP packets. Is there a diag command or something that will show that?

     

    ESP is it's own protocol , like tcp,udp,gre,icmp  ....and is not encapsulated into another protocol. IKSAKMP uses UDP.

     

    wiki or google ESP and protocol #50  or protocol #51 ( AH )

     

    That is my understanding.   TCP being 6 and UDP being 17.   Trying to leave no stone unturned here.  If someone was able to get max speed between a fiber peer and a coax peer I would love to know how.   Maybe these Comcast modems can not handle anything in volume other than IP 6 and 17.   I don't know

    Terms & ConditionsAccessibility statement