30E site-to-site VPN - slow, randomly erratic bandwidth

[train_wreck]

We have 2 30Es at separate locations. The main location is behind a 1gigabit symmetrical AT&T fiber line, the other is a 75/5 Mediacom. We are trying to get the full bandwidth from the main location to the remote site. Doing a regular iperf transfer from the ATT site to the remote site (no VPN) yields full bandwidth:

------------------------------------------------------------ Client connecting to 173.19.---.---, TCP port 5001 TCP window size: 64.0 KByte (default) ------------------------------------------------------------ [300] local 172.16.16.10 port 1363 connected with 173.19.---.--- port 5001 [ ID] Interval Transfer Bandwidth [300] 0.0- 1.0 sec 8.33 MBytes 69.9 Mbits/sec [300] 1.0- 2.0 sec 9.69 MBytes 81.3 Mbits/sec [300] 2.0- 3.0 sec 9.60 MBytes 80.5 Mbits/sec [300] 3.0- 4.0 sec 9.59 MBytes 80.5 Mbits/sec [300] 4.0- 5.0 sec 9.71 MBytes 81.5 Mbits/sec [300] 5.0- 6.0 sec 9.65 MBytes 80.9 Mbits/sec [300] 6.0- 7.0 sec 9.56 MBytes 80.2 Mbits/sec [300] 7.0- 8.0 sec 9.70 MBytes 81.3 Mbits/sec [300] 8.0- 9.0 sec 9.58 MBytes 80.3 Mbits/sec [300] 9.0-10.0 sec 9.60 MBytes 80.5 Mbits/sec [300] 0.0-10.2 sec 95.0 MBytes 78.1 Mbits/sec

We have now used the GUI wizard to create a "Site-to-site (Fortigate)" style IPsec VPN, with all defaults left as they are. When doing the same iperf test, we get very poor and inconsistent bandwidth:

------------------------------------------------------------ Client connecting to 192.168.192.2, TCP port 5001 TCP window size: 64.0 KByte (default) ------------------------------------------------------------ [324] local 172.16.16.10 port 1504 connected with 192.168.192.2 port 5001 [ ID] Interval Transfer Bandwidth [324] 0.0- 1.0 sec 2.92 MBytes 24.5 Mbits/sec [324] 1.0- 2.0 sec 1.24 MBytes 10.4 Mbits/sec [324] 2.0- 3.0 sec 2.84 MBytes 23.8 Mbits/sec [324] 3.0- 4.0 sec 3.55 MBytes 29.8 Mbits/sec [324] 4.0- 5.0 sec 4.11 MBytes 34.5 Mbits/sec [324] 5.0- 6.0 sec 4.24 MBytes 35.6 Mbits/sec [324] 6.0- 7.0 sec 4.89 MBytes 41.0 Mbits/sec [324] 7.0- 8.0 sec 4.69 MBytes 39.3 Mbits/sec [324] 8.0- 9.0 sec 4.84 MBytes 40.6 Mbits/sec [324] 9.0-10.0 sec 3.65 MBytes 30.6 Mbits/sec [324] 0.0-10.4 sec 37.0 MBytes 29.7 Mbits/sec

This is not just iperf; SCP and FTP file transfers have the exact same bandwidth level. Windows file copies are even worse, though I understand that is to be expected with Win SMB protocol. I understand there is an overhead with IPsec protocols, but this feels like something else.... VPN transfers are less than half non-VPN transfers.

Before sending the 30E to the remote site, I tested this by setting up a S2S VPN in the exact same way (using wizard) with both 30E WANs directly connected to each other, and I measured ~120mbps of performance.

So what is happening here? Apparently something along the path is slowing down our VPN. Is there anything we can do to get back the lost performance? CPU usage on either side never rises above ~2%, and mostly stays at 0. We have not configured any AV/inspection policies, only basic NAT firewall and VPN.

[rwpatterson]

Scratching my head as well, but thanks for that update.

[zandy]

Thanks for the update. I was testing a 30E and tried lowering the ciphers and nothing ...

Whenever I tried the Nat Traversal to "Force" I was able to max out on my VPN. weird bug.