DPDK acceleration for ipsec tunnels on virtual fortigates

[depereo]

We have been testing DPDK acceleration for ipsec tunnels on our nutanix hosts.

dpdk-iperf-1 and dpdk-iperf-2 are simple 4-core 8gb RAM ubuntu VMs with minor host tuning (sysctl window sizes etc) for iperf performance testing, and can maintain about 17Gbps using iperf3 when directly connected to one another.

dpdk-test-vm04-1 and dpdk-test-vm04-2 are fortigate VMs on v7.6.2 (2cpu, 16gb ram). When testing iperf performance on a basic ipsec tunnel we saw approximately 1.2Gbps between the ubuntu VMs. After enabling dpdk (see config below) we are only able to increase this performance to 1.5Gbps.

test diagram

I've confirmed that the dpdk engine is correctly picking up this traffic - ipsec_dec_packets and ipsec_enc_packets are incrementing, and the vnp and vnpsp engines all kick into life with `diagnose dpdk performance show` while the test is running.

However we did expect to see a significantly higher performance uplift for ipsec tunnels, is there something we're missing?

Current working DPDK config:
dpdk.global
    status=enable
    multiqueue=enable
    sleep-on-idle=enable
    elasticbuffer=enable
    per-session-accounting=1
    ipsec-offload=1
    hugepage-percentage=40
    nr_hugepages=3198
    mbufpool-percentage=30
    session-table-percentage=5
    protects=
dpdk.cpus
    en-cpus=(all) 0,1
    rx-cpus=(all) 0,1
    vnp-cpus=(all) 0,1
    vnpsp-cpus=(all) 0,1
    ips-cpus=(all) 0,1
    tx-cpus=(all) 0,1
    isolated-cpus=1
system.interface
    Interface "port1"
        dpdk=enable
    Interface "port2"
        dpdk=enable
    Interface "port3"
        dpdk=enable

[Anthony_E]

Hello,

Thank you for using the Community Forum. I will seek to get you an answer or help. We will reply to this thread with an update as soon as possible.

Thanks,

[Jean-Philippe_P]

Hello,

We are still looking for an answer to your question.

We will come back to you ASAP.

Thanks,

[atakannatak]

Hi @depereo ,

​You're observing limited performance gains from DPDK acceleration on your FortiGate VMs during IPsec throughput testing. Despite enabling DPDK and confirming that IPsec traffic is being offloaded, the throughput increase from approximately 1.2 Gbps to 1.5 Gbps suggests that additional optimizations may be necessary.​

• CPU Allocation and Affinity: Your current configuration assigns DPDK processes to CPUs 0 and 1. This limited allocation may not provide sufficient resources for optimal performance. Consider increasing the number of vCPUs and distributing DPDK processes across additional cores to enhance parallel processing capabilities.

         https://docs.fortinet.com/document/fortigate-private-cloud/7.6.0/kvm-administration-guide/79696/dpdk...

         https://docs.fortinet.com/document/fortigate-private-cloud/7.6.0/kvm-administration-guide/96687/isol...

• IPsec Soft Decryption Asynchronous Processing: In FortiOS versions prior to 7.4.2, enabling the ipsec-soft-dec-async setting can distribute IPsec session decryption across multiple CPU cores, potentially improving throughput. However, note that this setting has been removed in FortiOS versions 7.4.2 and later.

         https://community.fortinet.com/t5/FortiGate/Troubleshooting-Tip-Improving-IPsec-performance-throughp...

• Phase 1 and Phase 2 Proposal Optimization: Ensure that your IPsec Phase 1 and Phase 2 proposals utilize hardware-accelerated algorithms. For instance, AES256 is generally more efficient than 3DES and is supported by FortiGate's hardware acceleration features.​
• Host and VM Configuration: Some basic configurations might also be required on the virtualization side. These can be summarized as follows:

• Hugepages: Verify that the hugepage allocation is sufficient for your workload.​

         https://docs.fortinet.com/document/fortigate-private-cloud/7.6.0/vmware-esxi-administration-guide/73...

• NUMA Awareness: Ensure that vCPUs and memory are allocated within the same NUMA node to reduce latency.

         https://docs.fortinet.com/document/fortigate-private-cloud/7.6.0/vmware-esxi-administration-guide/49...

• SR-IOV and VT-d: If supported by your hardware and hypervisor, enabling SR-IOV and VT-d can provide direct I/O access, reducing overhead.​

         https://www.linkedin.com/pulse/boost-virtual-fortigate-firewall-performance-using-dpdk-joe-brunner-2...

• Virtio and RPS Optimization: FortiOS 7.6.1 introduces enhancements for IPsec performance through virtio and Receive Packet Steering (RPS), which can distribute packet processing across multiple CPUs.​

         https://fortinetweb.s3.amazonaws.com/docs.fortinet.com/v2/attachments/c8258039-3f9c-11ef-bfe5-fa163e...

         https://docs.fortinet.com/document/fortigate-private-cloud/7.6.0/kvm-administration-guide/758496/dpd...

If you’re still experiencing issues after following the steps provided in the reference links above, could you please run the following commands and share their output with us? This will help us better analyze the situation.

diagnose dpdk log show

diagnose dpdk statistics clear all

diagnose dpdk statistics show

diagnose dpdk performance show

diagnose sys mpstat 2 3

BR.

If my answer provided a solution for you, please mark the reply as solved it so that others can get it easily while searching for similar scenarios.

CCIE #68781

[abarushka]

Hello,

I would recommend to disable "sleep-on-idle" setting for performance reasons.

Moreover, I would recommend to check whether SR-IOV is enabled (all benchmarks are run when SR-IOV is enabled):

https://www.fortinet.com/content/dam/fortinet/assets/data-sheets/FortiGate_VM_KVM.pdf (page 7)

Also, I would recommend to check IPsec performance while AES256GCM cypher is set (as in the benchmark).

https://docs.fortinet.com/document/fortification/7.0.0/new-features/954778/add-ipsec-fast-path-in-vp... (list of supported cyphers)

[d_atella]

Based on your description, it seems the offload is not occurring. Check the npu_flag in the output of the diagnose vpn tunnel list command. For more details, refer to this knowledge base article:  Technical Tip: Ensuring IPSec traffic is offloaded for improved throughput