|
On affected firmware versions, FortiGates may permanently lose portions of their configuration if there is not enough free memory to store the current firewall configuration. Typically, this is associated with extremely low memory mode events. This can result in missing firewall policies, routing settings, VPN configurations, and other settings.
Since FortiGate models with 2GB of memory are more prone to memory exhaustion, this issue mostly affects lower-end devices.
To verify if the device experienced extreme low memory mode, use the command 'diagnose debug crashlog read' or review the System Events.
diagnose debug crashlog read
27: 2024-10-23 05:43:11 msg="Kernel enters extreme low memory mode"
28: 2024-10-23 05:43:12 MemTotal: 1963896 kB
29: 2024-10-23 05:43:12 MemFree: 84896 kB
30: 2024-10-23 05:43:12 Buffers: 208 kB
31: 2024-10-23 05:43:12 Cached: 428616 kB
32: 2024-10-23 05:43:12 SwapCached: 0 kB
33: 2024-10-23 05:43:12 Active: 1154704 kB
34: 2024-10-23 05:43:12 Inactive: 159332 kB
35: 2024-10-23 05:43:12 Active(anon): 1154080 kB
36: 2024-10-23 05:43:12 Inactive(anon): 159176 kB
37: 2024-10-23 05:43:12 Active(file): 624 kB
38: 2024-10-23 05:43:12 Inactive(file): 156 kB
39: 2024-10-23 05:43:12 Unevictable: 114908 kB
40: 2024-10-23 05:43:12 Mlocked: 44 kB
41: 2024-10-23 05:43:12 SwapTotal: 0 kB
42: 2024-10-23 05:43:12 SwapFree: 0 kB
43: 2024-10-23 05:43:12 Dirty: 0 kB
44: 2024-10-23 05:43:12 Writeback: 12 kB
45: 2024-10-23 05:43:12 AnonPages: 1000016 kB
46: 2024-10-23 05:43:12 Mapped: 232480 kB
47: 2024-10-23 05:43:12 Shmem: 313284 kB
48: 2024-10-23 05:43:12 Slab: 207452 kB
49: 2024-10-23 05:43:12 SReclaimable: 11264 kB
50: 2024-10-23 05:43:12 SUnreclaim: 196188 kB
51: 2024-10-23 05:43:12 KernelStack: 3520 kB
52: 2024-10-23 05:43:12 PageTables: 30056 kB
53: 2024-10-23 05:43:12 NFS_Unstable: 0 kB
54: 2024-10-23 05:43:12 Bounce: 0 kB
55: 2024-10-23 05:43:12 WritebackTmp: 0 kB
56: 2024-10-23 05:43:12 CommitLimit: 981948 kB
57: 2024-10-23 05:43:12 Committed_AS: 11752180 kB
58: 2024-10-23 05:43:12 VmallocTotal: 260046784 kB
59: 2024-10-23 05:43:12 VmallocUsed: 98872 kB
60: 2024-10-23 05:43:12 VmallocChunk: 259767452 kB
61: 2024-10-23 05:43:18 msg="Kernel exits extreme low memory mode"
62: 2024-10-23 05:43:18 service=kernel conserve=exit total="1917 MB" used="1524 MB" red="1687 MB"
63: 2024-10-23 05:43:18 green="1572 MB" msg="Kernel exits memory conserve mode"
System Events:
time=04:15:18 id=7506964253383852084 itime="2025-05-22 04:15:19" euid=3 epid=3 dsteuid=3 dstepid=3 logver=704072731 logid=0100022022 type="event" subtype="system" level="critical" msg="Kernel enters extreme low memory mode" logdesc="Extreme low memory mode entered"
Unlike system events stored in firewall memory, the crashlog is not removed after a firewall power cycle and can be used to confirm an extreme low memory event if system event logs are not available.
A frequent trigger for this issue has been identified as high memory usage during FortiGuard Antivirus updates, especially when running firmware v7.4.7 or v7.4.8 on lower-end devices.
It is strongly recommended to create a configuration backup immediately after any changes are made to the unit's configuration.
A periodical backup can also be stored with automation stitch configuration with FTP or TFTP, which is described in the following article: Technical Tip: How to send automated backups of the configuration from a FortiGate with an automatio...
The configuration loss issue has an identified fix, which is scheduled for inclusion in the following firmware versions:
These timelines for firmware release are estimates and may be subject to change. On firmware versions where this issue is resolved, the device does not lose portions of the configuration even if the device enters extreme low memory.
Workaround:
Optimize the FortiGate memory usage by applying the recommendations in the following KB article: Technical Tip: FortiGate is entering into Conserve Mode during FortiGuard Updates.
To collect the report in real time with automation stitch during conserve mode or high memory usage, the following article describes the process of report collection through email: Troubleshooting Tip: Collecting reports in memory conserve mode with automation stitch
Related article:
Technical Tip: Automated configuration backups with variable names based on the date
|
