|
FortiGate v7.0/v7.2 which has a big usage of log disk may encounter this issue.
To symptoms of the issue are as below:
- The freeable memory is high: run 'get sys perf status', the freeable memory is high but the free memory is low:
CPU states: 0% user 0% system 0% nice 100% idle 0% iowait 0% irq 0% softirq
CPU0 states: 3% user 0% system 0% nice 97% idle 0% iowait 0% irq 0% softirq
CPU1 states: 0% user 0% system 0% nice 100% idle 0% iowait 0% irq 0% softirq
CPU2 states: 0% user 0% system 0% nice 100% idle 0% iowait 0% irq 0% softirq
CPU3 states: 0% user 0% system 0% nice 100% idle 0% iowait 0% irq 0% softirq
CPU4 states: 0% user 0% system 0% nice 100% idle 0% iowait 0% irq 0% softirq
CPU5 states: 0% user 0% system 0% nice 100% idle 0% iowait 0% irq 0% softirq
CPU6 states: 0% user 0% system 0% nice 99% idle 0% iowait 0% irq 1% softirq
CPU7 states: 0% user 0% system 0% nice 100% idle 0% iowait 0% irq 0% softirq
CPU8 states: 0% user 2% system 0% nice 98% idle 0% iowait 0% irq 0% softirq
CPU9 states: 3% user 2% system 0% nice 95% idle 0% iowait 0% irq 0% softirq
CPU10 states: 0% user 0% system 0% nice 100% idle 0% iowait 0% irq 0% softirq
CPU11 states: 0% user 0% system 0% nice 100% idle 0% iowait 0% irq 0% softirq
CPU12 states: 0% user 0% system 0% nice 100% idle 0% iowait 0% irq 0% softirq
CPU13 states: 0% user 0% system 0% nice 100% idle 0% iowait 0% irq 0% softirq
CPU14 states: 1% user 0% system 0% nice 99% idle 0% iowait 0% irq 0% softirq
CPU15 states: 0% user 0% system 0% nice 100% idle 0% iowait 0% irq 0% softirq
Memory: 24719900k total, 14082504k used (57.0%), 1257460k free (5.1%), 9379936k freeable (37.9%)
Average network usage: 139556 / 139951 kbps in 1 minute, 58044 / 56614 kbps in 10 minutes, 60417 / 59402 kbps in 30 minutes
Maximal network usage: 362514 / 363194 kbps in 1 minute, 362514 / 363194 kbps in 10 minutes, 362514 / 363194 kbps in 30 minutes
Average sessions: 43362 sessions in 1 minute, 16319 sessions in 10 minutes, 13694 sessions in 30 minutes
Maximal sessions: 48070 sessions in 1 minute, 48070 sessions in 10 minutes, 48072 sessions in 30 minutes
Average session setup rate: 370 sessions per second in last 1 minute, 130 sessions per second in last 10 minutes, 58 sessions per second in last 30 minutes
Maximal session setup rate: 449 sessions per second in last 1 minute, 2154 sessions per second in last 10 minutes, 2154 sessions per second in last 30 minutes
Average NPU sessions: 12163 sessions in last 1 minute, 5590 sessions in last 10 minutes, 5266 sessions in last 30 minutes
Maximal NPU sessions: 13166 sessions in last 1 minute, 13166 sessions in last 10 minutes, 13166 sessions in last 30 minutes
Average nTurbo sessions: 57 sessions in last 1 minute, 42 sessions in last 10 minutes, 41 sessions in last 30 minutes
Maximal nTurbo sessions: 58 sessions in last 1 minute, 58 sessions in last 10 minutes, 58 sessions in last 30 minutes
Virus caught: 0 total in 1 minute
IPS attacks blocked: 0 total in 1 minute
Uptime: 115 days, 18 hours, 45 minutes
- The cache used up a huge memory: run 'diag hardware sysinfo memory', to see that the cached use up a lot of memory:
MemTotal: 24719900 kB
MemFree: 1267192 kB
MemAvailable: 12463952 kB
Buffers: 57004 kB
Cached: 12717320 kB
SwapCached: 0 kB
Active: 16330680 kB
Inactive: 1469664 kB
Active(anon): 6093868 kB
Inactive(anon): 242456 kB
Active(file): 10236812 kB
Inactive(file): 1227208 kB
Unevictable: 264224 kB
Mlocked: 0 kB
SwapTotal: 0 kB
SwapFree: 0 kB
Dirty: 180 kB
Writeback: 0 kB
AnonPages: 5290456 kB
Mapped: 706800 kB
Shmem: 1058620 kB
Slab: 845932 kB
SReclaimable: 142744 kB
SUnreclaim: 703188 kB
KernelStack: 6960 kB
PageTables: 133904 kB
NFS_Unstable: 0 kB
Bounce: 0 kB
WritebackTmp: 0 kB
CommitLimit: 12359948 kB
Committed_AS: 39976456 kB
VmallocTotal: 34359738367 kB
VmallocUsed: 0 kB
VmallocChunk: 0 kB
Percpu: 6848 kB
HardwareCorrupted: 0 kB
AnonHugePages: 0 kB
ShmemHugePages: 0 kB
ShmemPmdMapped: 0 kB
CmaTotal: 0 kB
CmaFree: 0 kB
DirectMap4k: 434176 kB
DirectMap2M: 20520960 kB
DirectMap1G: 6291456 kB
- If the freeable memory does not release the memory on time, and the FortiGate is out of memory, the FortiGate may not be able to process traffic.
Solution:
Upgrade to v7.0.14 or v7.2.8 or v7.4.4 and above.
Workarounds:
Option 1: Delete the disk logs (first backup the logs if needed):
FW3 # exec log filter device
Available devices:
0: memory
1: disk
2: fortianalyzer
3: fortianalyzer-cloud
4: forticloud
FW3 # exec log filter device 1
FW3 # exec log delete
This will delete disk traffic logs and all associated UTM logs.
Do you want to continue? (y/n)y
If this workaround succeeds, the 'freeable' memory (get sys perf status) should go down in a minute or two. Same for the 'Cached' memory in 'diag hardware sysinfo memory'.
Option 2:
The workaround is to move the logs to remote logging such as Syslog/FortiAnalyzer and disable disk logging with the steps below:
Step1: Disable disk logging:
config log disk setting
set status disable
end
Step 2: Format disk logging to clear out all the logs in FortiGate. Follow the below article to process:
Technical Tip: Standard procedure to format a FortiGate Log Disk, log backup from disk
|
