| Solution |
FortiGate v7.0/v7.2 which has a big usage of log disk may encounter this issue.
To symptoms of the issue are as below:
- The freeable memory is high: run 'get sys perf status', the freeable memory is high but the free memory is low:
CPU states: 0% user 0% system 0% nice 100% idle 0% iowait 0% irq 0% softirq
CPU0 states: 3% user 0% system 0% nice 97% idle 0% iowait 0% irq 0% softirq
CPU1 states: 0% user 0% system 0% nice 100% idle 0% iowait 0% irq 0% softirq
CPU2 states: 0% user 0% system 0% nice 100% idle 0% iowait 0% irq 0% softirq
CPU3 states: 0% user 0% system 0% nice 100% idle 0% iowait 0% irq 0% softirq
CPU4 states: 0% user 0% system 0% nice 100% idle 0% iowait 0% irq 0% softirq
CPU5 states: 0% user 0% system 0% nice 100% idle 0% iowait 0% irq 0% softirq
CPU6 states: 0% user 0% system 0% nice 99% idle 0% iowait 0% irq 1% softirq
CPU7 states: 0% user 0% system 0% nice 100% idle 0% iowait 0% irq 0% softirq
CPU8 states: 0% user 2% system 0% nice 98% idle 0% iowait 0% irq 0% softirq
CPU9 states: 3% user 2% system 0% nice 95% idle 0% iowait 0% irq 0% softirq
CPU10 states: 0% user 0% system 0% nice 100% idle 0% iowait 0% irq 0% softirq
CPU11 states: 0% user 0% system 0% nice 100% idle 0% iowait 0% irq 0% softirq
CPU12 states: 0% user 0% system 0% nice 100% idle 0% iowait 0% irq 0% softirq
CPU13 states: 0% user 0% system 0% nice 100% idle 0% iowait 0% irq 0% softirq
CPU14 states: 1% user 0% system 0% nice 99% idle 0% iowait 0% irq 0% softirq
CPU15 states: 0% user 0% system 0% nice 100% idle 0% iowait 0% irq 0% softirq
Memory: 24719900k total, 14082504k used (57.0%), 1257460k free (5.1%), 9379936k freeable (37.9%)
Average network usage: 139556 / 139951 kbps in 1 minute, 58044 / 56614 kbps in 10 minutes, 60417 / 59402 kbps in 30 minutes
Maximal network usage: 362514 / 363194 kbps in 1 minute, 362514 / 363194 kbps in 10 minutes, 362514 / 363194 kbps in 30 minutes
Average sessions: 43362 sessions in 1 minute, 16319 sessions in 10 minutes, 13694 sessions in 30 minutes
Maximal sessions: 48070 sessions in 1 minute, 48070 sessions in 10 minutes, 48072 sessions in 30 minutes
Average session setup rate: 370 sessions per second in last 1 minute, 130 sessions per second in last 10 minutes, 58 sessions per second in last 30 minutes
Maximal session setup rate: 449 sessions per second in last 1 minute, 2154 sessions per second in last 10 minutes, 2154 sessions per second in last 30 minutes
Average NPU sessions: 12163 sessions in last 1 minute, 5590 sessions in last 10 minutes, 5266 sessions in last 30 minutes
Maximal NPU sessions: 13166 sessions in last 1 minute, 13166 sessions in last 10 minutes, 13166 sessions in last 30 minutes
Average nTurbo sessions: 57 sessions in last 1 minute, 42 sessions in last 10 minutes, 41 sessions in last 30 minutes
Maximal nTurbo sessions: 58 sessions in last 1 minute, 58 sessions in last 10 minutes, 58 sessions in last 30 minutes
Virus caught: 0 total in 1 minute
IPS attacks blocked: 0 total in 1 minute
Uptime: 115 days, 18 hours, 45 minutes
- The cache used up a huge memory: run 'diag hardware sysinfo memory', to see that the cached use up a lot of memory:
MemTotal: 24719900 kB
MemFree: 1267192 kB
MemAvailable: 12463952 kB
Buffers: 57004 kB
Cached: 12717320 kB
SwapCached: 0 kB
Active: 16330680 kB
Inactive: 1469664 kB
Active(anon): 6093868 kB
Inactive(anon): 242456 kB
Active(file): 10236812 kB
Inactive(file): 1227208 kB
Unevictable: 264224 kB
Mlocked: 0 kB
SwapTotal: 0 kB
SwapFree: 0 kB
Dirty: 180 kB
Writeback: 0 kB
AnonPages: 5290456 kB
Mapped: 706800 kB
Shmem: 1058620 kB
Slab: 845932 kB
SReclaimable: 142744 kB
SUnreclaim: 703188 kB
KernelStack: 6960 kB
PageTables: 133904 kB
NFS_Unstable: 0 kB
Bounce: 0 kB
WritebackTmp: 0 kB
CommitLimit: 12359948 kB
Committed_AS: 39976456 kB
VmallocTotal: 34359738367 kB
VmallocUsed: 0 kB
VmallocChunk: 0 kB
Percpu: 6848 kB
HardwareCorrupted: 0 kB
AnonHugePages: 0 kB
ShmemHugePages: 0 kB
ShmemPmdMapped: 0 kB
CmaTotal: 0 kB
CmaFree: 0 kB
DirectMap4k: 434176 kB
DirectMap2M: 20520960 kB
DirectMap1G: 6291456 kB
- If the freeable memory does not release the memory on time, and the FortiGate is out of memory, the FortiGate may not be able to process traffic.
The workaround is to move the logs to remote logging such as SYSLOG/ FortiAnalyzer and disable disk logging with steps below:
Step1: Disable disk logging:
config log disk setting
set status disable
end
Step2: Format disk logging to clear out all the logs in FortiGate. Follow the below article to process:
Technical Tip: Standard procedure to format a FortiGate Log Disk, log backup from disk
Solution:
Upgrade to v7.0.14 or v7.2.8 or v7.4.4 and above.
|
