Strange/slow iperf on IPSEC tunnel LA <>NY 10gig

[liquidpower]

Hello, Im trying to find what my max throughput and then transfer Video files. 

I have a ha D3000 in LA and ha D3700 NYC both on 6.0.1

phase 1 aes128, sha1, Diffie hell groups 14 and 5

phase 2 aes128, sha1, replay detection enabled, perfect forward secrecy disabled 

both sites have 10gig circuits

I have 2 iperf box's connected straight in to the fg with 10g Ethernet  

current ping 0% packet loss round-trip min/avg/max = 69.2/69.2/69.3 ms

====

now if i do just a straight ieprf -c to my server in nyc i get  

iperf3.exe -c 192.168.205.2 -V -i 10
iperf 3.1.3
CYGWIN_NT-10.0 DESKTOP-05HP7RI 2.5.1(0.297/5/3) 2016-04-21 22:14 x86_64
Time: Wed, 14 Nov 2018 22:25:55 GMT
Connecting to host 192.168.205.2, port 5201
      Cookie: DESKTOP-05HP7RI.1542234355.450510.5a
      TCP MSS: 0 (default)
[ 4] local 192.168.220.2 port 57028 connected to 192.168.205.2 port 5201
Starting Test: protocol: TCP, 1 streams, 131072 byte blocks, omitting 0 seconds, 10 second test
[ ID] Interval Transfer Bandwidth
[ 4] 0.00-10.00 sec 26.2 MBytes 22.0 Mbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
Test Complete. Summary Results:
[ ID] Interval Transfer Bandwidth
[ 4] 0.00-10.00 sec 26.2 MBytes 22.0 Mbits/sec sender
[ 4] 0.00-10.00 sec 26.2 MBytes 22.0 Mbits/sec receiver
CPU Utilization: local/sender 1.7% (0.6%u/1.1%s), remote/receiver 0.0% (0.0%u/0.0%s)

if i try and lower the window size i see no dif

.\iperf3.exe -c 192.168.205.2 -V -i 10 -w k1300
iperf 3.1.3
CYGWIN_NT-10.0 DESKTOP-05HP7RI 2.5.1(0.297/5/3) 2016-04-21 22:14 x86_64
Time: Wed, 14 Nov 2018 22:48:37 GMT
Connecting to host 192.168.205.2, port 5201
      Cookie: DESKTOP-05HP7RI.1542235717.382544.3d
      TCP MSS: 0 (default)
[ 4] local 192.168.220.2 port 57051 connected to 192.168.205.2 port 5201
Starting Test: protocol: TCP, 1 streams, 131072 byte blocks, omitting 0 seconds, 10 second test
[ ID] Interval Transfer Bandwidth
[ 4] 0.00-10.02 sec 33.9 MBytes 28.4 Mbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
Test Complete. Summary Results:
[ ID] Interval Transfer Bandwidth
[ 4] 0.00-10.02 sec 33.9 MBytes 28.4 Mbits/sec sender
[ 4] 0.00-10.02 sec 33.9 MBytes 28.4 Mbits/sec receiver
CPU Utilization: local/sender 1.8% (0.8%u/1.1%s), remote/receiver 0.0% (0.0%u/0.0%s)

iperf Done.
.\iperf3.exe -c 192.168.205.2 -V -i 10 -w k1100
iperf 3.1.3
CYGWIN_NT-10.0 DESKTOP-05HP7RI 2.5.1(0.297/5/3) 2016-04-21 22:14 x86_64
Time: Wed, 14 Nov 2018 22:48:54 GMT
Connecting to host 192.168.205.2, port 5201
      Cookie: DESKTOP-05HP7RI.1542235734.037804.39
      TCP MSS: 0 (default)
[ 4] local 192.168.220.2 port 57053 connected to 192.168.205.2 port 5201
Starting Test: protocol: TCP, 1 streams, 131072 byte blocks, omitting 0 seconds, 10 second test
[ ID] Interval Transfer Bandwidth
[ 4] 0.00-10.01 sec 27.9 MBytes 23.4 Mbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
Test Complete. Summary Results:
[ ID] Interval Transfer Bandwidth
[ 4] 0.00-10.01 sec 27.9 MBytes 23.4 Mbits/sec sender
[ 4] 0.00-10.01 sec 27.8 MBytes 23.3 Mbits/sec receiver
CPU Utilization: local/sender 1.5% (0.6%u/0.9%s), remote/receiver 0.1% (0.0%u/0.1%s)

iperf Done.
 .\iperf3.exe -c 192.168.205.2 -V -i 10 -w k100
iperf 3.1.3
CYGWIN_NT-10.0 DESKTOP-05HP7RI 2.5.1(0.297/5/3) 2016-04-21 22:14 x86_64
Time: Wed, 14 Nov 2018 22:49:09 GMT
Connecting to host 192.168.205.2, port 5201
      Cookie: DESKTOP-05HP7RI.1542235749.688995.1a
      TCP MSS: 0 (default)
[ 4] local 192.168.220.2 port 57055 connected to 192.168.205.2 port 5201
Starting Test: protocol: TCP, 1 streams, 131072 byte blocks, omitting 0 seconds, 10 second test
[ ID] Interval Transfer Bandwidth
[ 4] 0.00-10.01 sec 32.6 MBytes 27.4 Mbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
Test Complete. Summary Results:
[ ID] Interval Transfer Bandwidth
[ 4] 0.00-10.01 sec 32.6 MBytes 27.4 Mbits/sec sender
[ 4] 0.00-10.01 sec 32.6 MBytes 27.4 Mbits/sec receiver
CPU Utilization: local/sender 2.0% (0.4%u/1.5%s), remote/receiver 0.2% (0.0%u/0.1%s)

my udp test

.\iperf3.exe -c 192.168.205.2 -u -b -v -i 10
Connecting to host 192.168.205.2, port 5201
[ 4] local 192.168.220.2 port 61537 connected to 192.168.205.2 port 5201
[ ID] Interval Transfer Bandwidth Total Datagrams
[ 4] 0.00-10.00 sec 5.59 GBytes 4.80 Gbits/sec 732220
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bandwidth Jitter Lost/Total Datagrams
[ 4] 0.00-10.00 sec 5.59 GBytes 4.80 Gbits/sec 0.016 ms 330290/731737 (45%)
[ 4] Sent 731737 datagrams

and last thing i tried was adjusting the mss but i get no change from that as well

(do i need to set that on the server side as well?)

.\iperf3.exe -c 192.168.205.2 -V -i 10 -M 1024k
iperf 3.1.3
CYGWIN_NT-10.0 DESKTOP-05HP7RI 2.5.1(0.297/5/3) 2016-04-21 22:14 x86_64
Time: Wed, 14 Nov 2018 23:01:57 GMT
Connecting to host 192.168.205.2, port 5201
      Cookie: DESKTOP-05HP7RI.1542236517.459028.3a
      TCP MSS: 1024
[ 4] local 192.168.220.2 port 57073 connected to 192.168.205.2 port 5201
Starting Test: protocol: TCP, 1 streams, 131072 byte blocks, omitting 0 seconds, 10 second test
[ ID] Interval Transfer Bandwidth
[ 4] 0.00-10.01 sec 33.0 MBytes 27.7 Mbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
Test Complete. Summary Results:
[ ID] Interval Transfer Bandwidth
[ 4] 0.00-10.01 sec 33.0 MBytes 27.7 Mbits/sec sender
[ 4] 0.00-10.01 sec 33.0 MBytes 27.7 Mbits/sec receiver
CPU Utilization: local/sender 2.1% (1.1%u/1.1%s), remote/receiver 0.0% (0.0%u/0.0%s)

and lastly if i do 80 streams at once i get like 901Mbits per sec. i checked the tcp throughput calculator and with a rtt of 69 my max should be around 1692.75 Mbit/sec. so what gives?

any help would be huge. thanks so much!