Conserve mode

[AEK]

Hello

FGT 1801F with FOS 7.0.12.

Please see the below output and confirm if this is a conserve/extreme mode condition, knowing that at the same time my FGT started to reject sessions.

In case the below is conserve mode condition, what can be the reasons for which a FortiGate doesn''t log that the system has entered conserve mode?

# get system performance status
...
Memory: 24719900k total, 12677632k used (51.3%), 1229868k free (5.0%), 10812400k freeable (43.7%)
...

# diagnose hardware sysinfo memory
MemTotal: 24719900 kB
MemFree: 1228440 kB
MemAvailable: 14488404 kB
Buffers: 821188 kB
Cached: 12821328 kB
SwapCached: 0 kB
Active: 14187032 kB
Inactive: 1372760 kB
Active(anon): 2216820 kB
Inactive(anon): 58332 kB
Active(file): 11970212 kB
Inactive(file): 1314428 kB
Unevictable: 241912 kB
Mlocked: 0 kB
SwapTotal: 0 kB
SwapFree: 0 kB
Dirty: 2720 kB
Writeback: 0 kB
AnonPages: 2160024 kB
Mapped: 252804 kB
Shmem: 124376 kB
Slab: 3250364 kB
SReclaimable: 428828 kB
SUnreclaim: 2821536 kB
KernelStack: 6768 kB
PageTables: 100536 kB
NFS_Unstable: 0 kB
Bounce: 0 kB
WritebackTmp: 0 kB
CommitLimit: 12359948 kB
Committed_AS: 34108440 kB
VmallocTotal: 34359738367 kB
VmallocUsed: 0 kB
VmallocChunk: 0 kB
Percpu: 7808 kB
HardwareCorrupted: 0 kB
AnonHugePages: 0 kB
ShmemHugePages: 0 kB
ShmemPmdMapped: 0 kB
CmaTotal: 0 kB
CmaFree: 0 kB
DirectMap4k: 114688 kB
DirectMap2M: 11403264 kB
DirectMap1G: 15728640 kB

[dbu]

Hi @AEK ,

From the output provided it looks like you are using only half of the memory.

Please share output from this command : 

# diagnose hardware sysinfo conserve
memory conserve mode: on
total RAM: 997 MB
memory used: 735 MB 73% of total RAM
memory freeable: 173 MB 17% of total RAM
memory used + freeable threshold extreme: 947 MB 95% of total RAM
memory used threshold red: 877 MB 88% of total RAM
memory used threshold green: 817 MB 82% of total RAM

Regards!

[AEK]

Hi Dbu

Unfortunately I didn't check this command at that time because no message was logged regarding conserve mode, so at that time I thought that the system was not in conserve mode.

However I think that memory for conserve mode is calculated with used+freeable, right? Because if I understand well the freeable memory is cache & buffers that are actually used by the system.

We can see this in an old output from another 1801F here below, where we can see that the system is in conserve mode while the used mem is 40%.

# get sys perf stat
...
Memory: 24719848k total, 9886572k used (40.0%), 3034828k free (12.3%), 11798448k freeable (47.7%)
...


# diagnose hard sys conserve
memory conserve mode: on
total RAM: 24140 MB
memory used: 9637 MB 39% of total RAM
memory freeable: 11445 MB 47% of total RAM
memory used + freeable threshold extreme: 22933 MB 95% of total RAM
memory used threshold red: 21243 MB 88% of total RAM
memory used threshold green: 19794 MB 82% of total RAM


# diag hard sysinfo memory 
MemTotal: 24719848 kB
MemFree: 3093076 kB
MemAvailable: 17216092 kB
Buffers: 24960 kB
Cached: 15355516 kB
SwapCached: 0 kB
Active: 15030296 kB
Inactive: 1678052 kB
Active(anon): 1810800 kB
Inactive(anon): 475868 kB
Active(file): 13219496 kB
Inactive(file): 1202184 kB
Unevictable: 271356 kB
Mlocked: 0 kB
SwapTotal: 0 kB
SwapFree: 0 kB
Dirty: 0 kB
Writeback: 0 kB
AnonPages: 1599408 kB
Mapped: 322900 kB
Shmem: 693992 kB
Slab: 358668 kB
SReclaimable: 80160 kB
SUnreclaim: 278508 kB
KernelStack: 6672 kB
PageTables: 101052 kB
NFS_Unstable: 0 kB
Bounce: 0 kB
WritebackTmp: 0 kB
CommitLimit: 12359924 kB
Committed_AS: 33501680 kB
VmallocTotal: 34359738367 kB
VmallocUsed: 0 kB
VmallocChunk: 0 kB
Percpu: 6976 kB
HardwareCorrupted: 0 kB
AnonHugePages: 0 kB
ShmemHugePages: 0 kB
ShmemPmdMapped: 0 kB
CmaTotal: 0 kB
CmaFree: 0 kB
DirectMap4k: 38912 kB
DirectMap2M: 6236160 kB
DirectMap1G: 20971520 kB

[srajeswaran]

This scenario the conserve mode won't be triggered even though the memory usage is 95% (only 5% is free). The reason is your memory is used by freeable (43.7%), which is not part of conserve mode calculation. Ideally the freeable memory is free and can be utilized by system when there is a requirement ,but we have identified a bug -  when disk logging is enabled the freeable memory usage may go up and result in unexpected behaviors (882187 and 918571). The fix is available in 7.0.13 (7.0.12 has a partial fix) or the workaround is to disable disk logging.

[AEK]

Hello Suraj

Thanks for the information. In fact it is probable that my FG is suffering of the same symptome.
However now I'm confused with when you said freeable mem is not used in conserve/extreme mode calculation.
You can see in the below examples on another FG 1801F that conserve mode is triggered when "used" mem is even below 50%.
So as per my understanding I think the freeable mem (or at least a portion of it) is also used in calculation, since it contains some part of cache and buffers that are actively used by the system and can't be simply released on demand.
Please correct me if I'm wrong.

887: 2022-09-01 09:40:24 service=kernel conserve=on total="24140 MB" used="10498 MB" red="21243 MB"
888: 2022-09-01 09:40:24 green="19794 MB" msg="Kernel enters memory conserve mode"
889: 2022-09-01 09:40:24 MemTotal: 24719848 kB
890: 2022-09-01 09:40:24 MemFree: 2962716 kB
891: 2022-09-01 09:40:24 MemAvailable: 16248652 kB
892: 2022-09-01 09:40:24 Buffers: 261756 kB
893: 2022-09-01 09:40:24 Cached: 14547276 kB
894: 2022-09-01 09:40:24 SwapCached: 0 kB
895: 2022-09-01 09:40:24 Active: 14237168 kB
896: 2022-09-01 09:40:24 Inactive: 2439292 kB
897: 2022-09-01 09:40:24 Active(anon): 2665656 kB
898: 2022-09-01 09:40:24 Inactive(anon): 493548 kB
899: 2022-09-01 09:40:24 Active(file): 11571512 kB
900: 2022-09-01 09:40:24 Inactive(file): 1945744 kB


979: 2022-09-01 10:49:10 service=kernel conserve=on total="24140 MB" used="9649 MB" red="21243 MB"
980: 2022-09-01 10:49:10 green="19794 MB" msg="Kernel enters memory conserve mode"
981: 2022-09-01 10:49:10 MemTotal: 24719848 kB
982: 2022-09-01 10:49:10 MemFree: 2964668 kB
983: 2022-09-01 10:49:10 MemAvailable: 17238804 kB
984: 2022-09-01 10:49:10 Buffers: 32932 kB
985: 2022-09-01 10:49:10 Cached: 15502632 kB
986: 2022-09-01 10:49:10 SwapCached: 0 kB
987: 2022-09-01 10:49:10 Active: 14661448 kB
988: 2022-09-01 10:49:10 Inactive: 2191896 kB
989: 2022-09-01 10:49:10 Active(anon): 1789064 kB
990: 2022-09-01 10:49:10 Inactive(anon): 487448 kB
991: 2022-09-01 10:49:10 Active(file): 12872384 kB
992: 2022-09-01 10:49:10 Inactive(file): 1704448 kB


1168: 2022-10-14 10:44:04 service=kernel conserve=on total="24140 MB" used="11051 MB" red="22933 MB"
1169: 2022-10-14 10:44:04 green="21726 MB" msg="Kernel enters memory conserve mode"
1170: 2022-10-14 10:44:04 MemTotal: 24719848 kB
1171: 2022-10-14 10:44:04 MemFree: 1234884 kB
1172: 2022-10-14 10:44:04 MemAvailable: 16097352 kB
1173: 2022-10-14 10:44:04 Buffers: 726660 kB
1174: 2022-10-14 10:44:04 Cached: 15575288 kB
1175: 2022-10-14 10:44:04 SwapCached: 0 kB
1176: 2022-10-14 10:44:04 Active: 16289308 kB
1177: 2022-10-14 10:44:04 Inactive: 1987816 kB
1178: 2022-10-14 10:44:04 Active(anon): 2840116 kB
1179: 2022-10-14 10:44:04 Inactive(anon): 473096 kB
1180: 2022-10-14 10:44:04 Active(file): 13449192 kB
1181: 2022-10-14 10:44:04 Inactive(file): 1514720 kB


350: 2022-09-09 09:57:16 service=kernel conserve=on total="24140 MB" used="11071 MB" red="22933 MB" 
351: 2022-09-09 09:57:16 green="21726 MB" msg="Kernel enters memory conserve mode"
352: 2022-09-09 09:57:17 MemTotal: 24719848 kB
353: 2022-09-09 09:57:17 MemFree: 1235600 kB
354: 2022-09-09 09:57:17 MemAvailable: 16023224 kB
355: 2022-09-09 09:57:17 Buffers: 539112 kB
356: 2022-09-09 09:57:17 Cached: 15736824 kB
357: 2022-09-09 09:57:17 SwapCached: 0 kB
358: 2022-09-09 09:57:17 Active: 16883340 kB
359: 2022-09-09 09:57:17 Inactive: 1416028 kB
360: 2022-09-09 09:57:17 Active(anon): 2869300 kB
361: 2022-09-09 09:57:17 Inactive(anon): 489040 kB
362: 2022-09-09 09:57:17 Active(file): 14014040 kB
363: 2022-09-09 09:57:17 Inactive(file): 926988 kB


443: 2022-09-13 09:21:16 service=kernel conserve=on total="24140 MB" used="9984 MB" red="22933 MB" 
444: 2022-09-13 09:21:16 green="21726 MB" msg="Kernel enters memory conserve mode"
445: 2022-09-13 09:21:16 MemTotal: 24719848 kB
446: 2022-09-13 09:21:16 MemFree: 1235444 kB
447: 2022-09-13 09:21:16 MemAvailable: 17261224 kB
448: 2022-09-13 09:21:16 Buffers: 141636 kB
449: 2022-09-13 09:21:16 Cached: 17210688 kB
450: 2022-09-13 09:21:16 SwapCached: 0 kB
451: 2022-09-13 09:21:16 Active: 16342428 kB
452: 2022-09-13 09:21:16 Inactive: 2213932 kB
453: 2022-09-13 09:21:16 Active(anon): 1766628 kB
454: 2022-09-13 09:21:16 Inactive(anon): 476120 kB
455: 2022-09-13 09:21:16 Active(file): 14575800 kB
456: 2022-09-13 09:21:16 Inactive(file): 1737812 kB

[srajeswaran · Accepted solution]

Can you confirm if the above conserve mode log examples are from 7.0.12 or an older version? On 7.0.7 and some other older versions due to bug - 798303, conserve mode was triggered even when the usage is lower than the red threshold .
But this issue is fixed from 7.0.8 , so I don't expect you to see these logs on device running versions above 7.0.8 (or these logs should be generated before upgrade).

Regarding the calculation of conserve mode, conserve mode memory calculation don't include the freeable memory.  The logs that you are shared are a buggy behavior, I hope this clarifies.

Can you confirm if you have disk logging enabled? If so, you may need to disable it for now.

[AEK]

Yes I confirm it is old FOS (7.0.6).

Yes my disg logging is enabled and I'll disable it for now.

Thanks you very much for the explanation. Many things are clear now.