|
When troubleshooting CPU issues, the first step is to determine in which category of usage it is. This case focuses on high system space usage.
The following is an example of high system usage:
FGT01 # get system performance status
CPU states: 0% user 99% system 0% nice 1% idle 0% iowait 0% irq 0% softirq
CPU0 states: 0% user 99% system 0% nice 1% idle 0% iowait 0% irq 0% softirq
CPU1 states: 0% user 98% system 0% nice 2% idle 0% iowait 0% irq 0% softirq
Since this issue can happen very intermittently, it is best to use diagnose sys mpstat instead of diagnose system performance status.
FGT01 # diagnose sys mpstat
TIME CPU %usr %nice %sys %iowait %irq %soft %steal %idle
02:24:14 PM all 7.88 0.00 61.48 0.00 0.10 2.24 0.00 28.31
0 8.95 0.00 60.04 0.00 0.20 1.39 0.00 29.42
1 7.75 0.00 61.43 0.00 0.00 1.99 0.00 28.83
2 8.75 0.00 56.26 0.00 0.20 2.19 0.00 32.60
3 13.12 0.00 56.06 0.00 0.00 2.39 0.00 28.43
4 7.75 0.00 62.43 0.00 0.20 1.79 0.00 27.83
5 7.55 0.00 61.83 0.00 0.00 1.79 0.00 28.83
6 5.57 0.00 66.60 0.00 0.20 2.98 0.00 24.65
7 3.58 0.00 67.20 0.00 0.00 3.38 0.00 25.84
TIME CPU %usr %nice %sys %iowait %irq %soft %steal %idle
02:24:19 PM all 6.54 0.00 90.04 0.00 0.00 2.59 0.00 0.82
0 8.38 0.00 87.03 0.00 0.00 2.59 0.00 2.00
1 8.98 0.00 87.82 0.00 0.00 2.00 0.00 1.20
2 5.19 0.00 92.42 0.00 0.00 2.20 0.00 0.20
3 5.39 0.00 90.22 0.00 0.00 3.39 0.00 1.00
4 9.38 0.00 88.42 0.00 0.00 2.20 0.00 0.00
5 5.19 0.00 91.22 0.00 0.00 3.19 0.00 0.40
6 7.39 0.00 88.82 0.00 0.00 2.59 0.00 1.20
7 2.40 0.00 94.41 0.00 0.00 2.59 0.00 0.60
Once high system space is identified, perform CPU profiling. This shows what functions are being called by FortiOS and can provide some insight why high system space usage has been observed. Follow the steps provided in Troubleshooting Tip: FortiGate CPU Profiling.
In cases of source port exhaustion, the function inet_hash_connect will be found in the list. The higher it is in the list, the more FortiOS is preoccupied with this issue.
Kernel=0xffffffff80200190-0xffffffff8062c69d, module-gap=0x1f9d3963
Profile buffer: profile step=4, sz=6563730-3281865, last-addr=0xffffffffa0858a17
warning: functions cannot be profiled properly, early_idt_handlers-_einittext
0xffffffff80556ef8: 658 inet_hash_connect+0x38/0x40
0xffffffff805562d4: 38 __inet_check_established+0xf4/0x2d0
Possible workarounds:
- Option 1: increase the source port range
config system global
set ip-src-port-range 1024-65535 (default is 25000)
end
The issue with this option is that increasing the source port range will cause potential session clashes as both the proxy and source NAT can potentially use the same ports.
There are two layers that may allocate port for a TCP session:
- TCP layer - When proxy connects to the server, typically using outgoing interface IP;
- Firewall layer - source NAT, typically also using the outgoing interface IP;
The configurable port range is for TCP layer. By default, it is not overlapping with the port range(fixed, not configurable) used by firewall layer. When enlarging the configurable port range the two port ranges start to overlap each other.
Having multiple outgoing IPs will increase the source ports FortiOS can draw from by 25000 for every added IP.
The following is an example configuration:
config system interface
edit "vlan_1"
set vdom "root"
set ip 194.11.1.250 255.255.255.0
set device-identification enable
set role lan
set snmp-index 59
set secondary-IP enable
set interface "port1"
set vlanid 1
config secondaryip
edit 1
set ip 194.11.1.251 255.255.255.0
next
edit 2
set ip 194.11.1.252 255.255.255.0
next
edit 3
set ip 194.11.1.253 255.255.255.0
next
end
next
end
config web-proxy explicit
set status enable
set http-incoming-port 8080
set outgoing-ip 194.11.1.251 194.11.1.252 194.11.1.253
end
A lower session TTL will free up ports more quickly. For more information, refer to the following articles:
Technical Tip: Change session TTL on firewall policy
Technical Tip: Default session timeout value (session-ttl)
|
