| Solution |
Symptoms:
- HTTP client sessions fail or reset unexpectedly.
- Performance testing shows timeouts, even when CPU usage on FGVM appears normal.
- TCP captures indicate missing FIN packets, retransmissions, and keep-alive probes.
- DPDK engine CPU cores often run at full usage due to polling mode.
- NAT/proxy configurations may contribute to session anomalies.
- This is for AT&T Flexware Service running on Juniper CPE device with FortiGate VM.
Possible Causes:
- CPU Saturation: DPDK polling mode consumes all CPU cycles for Rx/Tx cores, saturating CPU when UTM functions are enabled.
- Juniper NFX350 Hardware Limitations: Rx/Tx share CPUs with VNP and IPS, multi-queue is not fully supported.
- UTM offload limitations: DPDK offloads firewall and IPS only; AV, Web Filter, WAD, and logging remain on kernel CPU.
- Packet Drops at OVS Level: Packet drops can occur in the Open vSwitch on the host.
- Test client behavior: No FIN packets can cause server-side RSTs.
Diagnostic Findings:
-
No drops are detected in the FGVM debug logs.
-
Engine cores show high CPU utilization, with some cores having 15–25% idle time.
-
TCP Packet Capture Analysis:
-
Server sends FIN, but client fails to respond properly.
-
Keep alive packets sent by server; eventually, server sends RST to terminate the session.
-
NAT is enabled on the system.
config dpdk cpus
set rx-cpus "13,15"
set vnp-cpus "0-11"
set ips-cpus "0-11"
set tx-cpus "12,14"
set isolated-cpus "12-15"
end
- DPDK Rx/Tx reserved CPUs.
- Kernel CPUs separate for UTM processing.
Workaround:
CPU Affinity Tuning:
set rx-cpus "8-15"
set vnp-cpus "8-15"
set ips-cpus "8-15"
set tx-cpus "8-15"
set isolated-cpus "8-15"
- Reserve dedicated CPU cores for DPDK Rx/Tx.
- Separate kernel and UTM functions on different cores.
- Adjust CPU allocation for AV, Web Filter, WAD, logging, and IPS.
Optimize UTM Feature Affinity:
config system global
set av-affinity "0"
set url-filter-affinity "0"
set wad-affinity "0"
set miglog-affinity "0"
set syslog-affinity "0"
end
- Reduces CPU contention between DPDK and UTM services.
DPDK Global Settings:
config dpdk global
set status enable
set interface "port1" "port2" "port3"
set multiqueue enable
set sleep-on-idle enable
end
- 'sleep-on-idle enable' can reduce CPU usage.
- DPDK works in polling mode: tuning can save CPU resources.
|
