Skip to main content
AEK
SuperUser
AEK
SuperUser
October 7, 2023
Solved

Conserve mode

  • October 7, 2023
  • 6 replies
  • 4887 views

Hello

FGT 1801F with FOS 7.0.12.

Please see the below output and confirm if this is a conserve/extreme mode condition, knowing that at the same time my FGT started to reject sessions.

In case the below is conserve mode condition, what can be the reasons for which a FortiGate doesn''t log that the system has entered conserve mode?

 

# get system performance status
...
Memory: 24719900k total, 12677632k used (51.3%), 1229868k free (5.0%), 10812400k freeable (43.7%)
...

 

# diagnose hardware sysinfo memory
MemTotal: 24719900 kB
MemFree: 1228440 kB
MemAvailable: 14488404 kB
Buffers: 821188 kB
Cached: 12821328 kB
SwapCached: 0 kB
Active: 14187032 kB
Inactive: 1372760 kB
Active(anon): 2216820 kB
Inactive(anon): 58332 kB
Active(file): 11970212 kB
Inactive(file): 1314428 kB
Unevictable: 241912 kB
Mlocked: 0 kB
SwapTotal: 0 kB
SwapFree: 0 kB
Dirty: 2720 kB
Writeback: 0 kB
AnonPages: 2160024 kB
Mapped: 252804 kB
Shmem: 124376 kB
Slab: 3250364 kB
SReclaimable: 428828 kB
SUnreclaim: 2821536 kB
KernelStack: 6768 kB
PageTables: 100536 kB
NFS_Unstable: 0 kB
Bounce: 0 kB
WritebackTmp: 0 kB
CommitLimit: 12359948 kB
Committed_AS: 34108440 kB
VmallocTotal: 34359738367 kB
VmallocUsed: 0 kB
VmallocChunk: 0 kB
Percpu: 7808 kB
HardwareCorrupted: 0 kB
AnonHugePages: 0 kB
ShmemHugePages: 0 kB
ShmemPmdMapped: 0 kB
CmaTotal: 0 kB
CmaFree: 0 kB
DirectMap4k: 114688 kB
DirectMap2M: 11403264 kB
DirectMap1G: 15728640 kB

 

 

Best answer by srajeswaran

Can you confirm if the above conserve mode log examples are from 7.0.12 or an older version? On 7.0.7 and some other older versions due to bug - 798303, conserve mode was triggered even when the usage is lower than the red threshold .
But this issue is fixed from 7.0.8 , so I don't expect you to see these logs on device running versions above 7.0.8 (or these logs should be generated before upgrade).

Regarding the calculation of conserve mode, conserve mode memory calculation don't include the freeable memory.  The logs that you are shared are a buggy behavior, I hope this clarifies.

Can you confirm if you have disk logging enabled? If so, you may need to disable it for now.

6 replies

dbu
Staff
dbu
Staff
October 7, 2023

Hi @AEK ,


From the output provided it looks like you are using only half of the memory.

 

Please share output from this command : 

 

# diagnose hardware sysinfo conserve
memory conserve mode: on
total RAM: 997 MB
memory used: 735 MB 73% of total RAM
memory freeable: 173 MB 17% of total RAM
memory used + freeable threshold extreme: 947 MB 95% of total RAM
memory used threshold red: 877 MB 88% of total RAM
memory used threshold green: 817 MB 82% of total RAM

 

Regards!

AEK
SuperUser
AEKAuthor
SuperUser
October 8, 2023

Hi Dbu

Unfortunately I didn't check this command at that time because no message was logged regarding conserve mode, so at that time I thought that the system was not in conserve mode.

However I think that memory for conserve mode is calculated with used+freeable, right? Because if I understand well the freeable memory is cache & buffers that are actually used by the system.

 

We can see this in an old output from another 1801F here below, where we can see that the system is in conserve mode while the used mem is 40%.

 

# get sys perf stat
...
Memory: 24719848k total, 9886572k used (40.0%), 3034828k free (12.3%), 11798448k freeable (47.7%)
...


# diagnose hard sys conserve
memory conserve mode: on
total RAM: 24140 MB
memory used: 9637 MB 39% of total RAM
memory freeable: 11445 MB 47% of total RAM
memory used + freeable threshold extreme: 22933 MB 95% of total RAM
memory used threshold red: 21243 MB 88% of total RAM
memory used threshold green: 19794 MB 82% of total RAM


# diag hard sysinfo memory 
MemTotal: 24719848 kB
MemFree: 3093076 kB
MemAvailable: 17216092 kB
Buffers: 24960 kB
Cached: 15355516 kB
SwapCached: 0 kB
Active: 15030296 kB
Inactive: 1678052 kB
Active(anon): 1810800 kB
Inactive(anon): 475868 kB
Active(file): 13219496 kB
Inactive(file): 1202184 kB
Unevictable: 271356 kB
Mlocked: 0 kB
SwapTotal: 0 kB
SwapFree: 0 kB
Dirty: 0 kB
Writeback: 0 kB
AnonPages: 1599408 kB
Mapped: 322900 kB
Shmem: 693992 kB
Slab: 358668 kB
SReclaimable: 80160 kB
SUnreclaim: 278508 kB
KernelStack: 6672 kB
PageTables: 101052 kB
NFS_Unstable: 0 kB
Bounce: 0 kB
WritebackTmp: 0 kB
CommitLimit: 12359924 kB
Committed_AS: 33501680 kB
VmallocTotal: 34359738367 kB
VmallocUsed: 0 kB
VmallocChunk: 0 kB
Percpu: 6976 kB
HardwareCorrupted: 0 kB
AnonHugePages: 0 kB
ShmemHugePages: 0 kB
ShmemPmdMapped: 0 kB
CmaTotal: 0 kB
CmaFree: 0 kB
DirectMap4k: 38912 kB
DirectMap2M: 6236160 kB
DirectMap1G: 20971520 kB

 

 

AEK
srajeswaran
Staff
srajeswaran
Staff
October 8, 2023

This scenario the conserve mode won't be triggered even though the memory usage is 95% (only 5% is free). The reason is your memory is used by freeable (43.7%), which is not part of conserve mode calculation. Ideally the freeable memory is free and can be utilized by system when there is a requirement ,but we have identified a bug -  when disk logging is enabled the freeable memory usage may go up and result in unexpected behaviors (882187 and 918571). The fix is available in 7.0.13 (7.0.12 has a partial fix) or the workaround is to disable disk logging.

    AEK
    SuperUser
    AEKAuthor
    SuperUser
    October 8, 2023

    Hello Suraj

    Thanks for the information. In fact it is probable that my FG is suffering of the same symptome.
    However now I'm confused with when you said freeable mem is not used in conserve/extreme mode calculation.
    You can see in the below examples on another FG 1801F that conserve mode is triggered when "used" mem is even below 50%.
    So as per my understanding I think the freeable mem (or at least a portion of it) is also used in calculation, since it contains some part of cache and buffers that are actively used by the system and can't be simply released on demand.
    Please correct me if I'm wrong.

     

    887: 2022-09-01 09:40:24 service=kernel conserve=on total="24140 MB" used="10498 MB" red="21243 MB"
    888: 2022-09-01 09:40:24 green="19794 MB" msg="Kernel enters memory conserve mode"
    889: 2022-09-01 09:40:24 MemTotal: 24719848 kB
    890: 2022-09-01 09:40:24 MemFree: 2962716 kB
    891: 2022-09-01 09:40:24 MemAvailable: 16248652 kB
    892: 2022-09-01 09:40:24 Buffers: 261756 kB
    893: 2022-09-01 09:40:24 Cached: 14547276 kB
    894: 2022-09-01 09:40:24 SwapCached: 0 kB
    895: 2022-09-01 09:40:24 Active: 14237168 kB
    896: 2022-09-01 09:40:24 Inactive: 2439292 kB
    897: 2022-09-01 09:40:24 Active(anon): 2665656 kB
    898: 2022-09-01 09:40:24 Inactive(anon): 493548 kB
    899: 2022-09-01 09:40:24 Active(file): 11571512 kB
    900: 2022-09-01 09:40:24 Inactive(file): 1945744 kB


    979: 2022-09-01 10:49:10 service=kernel conserve=on total="24140 MB" used="9649 MB" red="21243 MB"
    980: 2022-09-01 10:49:10 green="19794 MB" msg="Kernel enters memory conserve mode"
    981: 2022-09-01 10:49:10 MemTotal: 24719848 kB
    982: 2022-09-01 10:49:10 MemFree: 2964668 kB
    983: 2022-09-01 10:49:10 MemAvailable: 17238804 kB
    984: 2022-09-01 10:49:10 Buffers: 32932 kB
    985: 2022-09-01 10:49:10 Cached: 15502632 kB
    986: 2022-09-01 10:49:10 SwapCached: 0 kB
    987: 2022-09-01 10:49:10 Active: 14661448 kB
    988: 2022-09-01 10:49:10 Inactive: 2191896 kB
    989: 2022-09-01 10:49:10 Active(anon): 1789064 kB
    990: 2022-09-01 10:49:10 Inactive(anon): 487448 kB
    991: 2022-09-01 10:49:10 Active(file): 12872384 kB
    992: 2022-09-01 10:49:10 Inactive(file): 1704448 kB


    1168: 2022-10-14 10:44:04 service=kernel conserve=on total="24140 MB" used="11051 MB" red="22933 MB"
    1169: 2022-10-14 10:44:04 green="21726 MB" msg="Kernel enters memory conserve mode"
    1170: 2022-10-14 10:44:04 MemTotal: 24719848 kB
    1171: 2022-10-14 10:44:04 MemFree: 1234884 kB
    1172: 2022-10-14 10:44:04 MemAvailable: 16097352 kB
    1173: 2022-10-14 10:44:04 Buffers: 726660 kB
    1174: 2022-10-14 10:44:04 Cached: 15575288 kB
    1175: 2022-10-14 10:44:04 SwapCached: 0 kB
    1176: 2022-10-14 10:44:04 Active: 16289308 kB
    1177: 2022-10-14 10:44:04 Inactive: 1987816 kB
    1178: 2022-10-14 10:44:04 Active(anon): 2840116 kB
    1179: 2022-10-14 10:44:04 Inactive(anon): 473096 kB
    1180: 2022-10-14 10:44:04 Active(file): 13449192 kB
    1181: 2022-10-14 10:44:04 Inactive(file): 1514720 kB


    350: 2022-09-09 09:57:16 service=kernel conserve=on total="24140 MB" used="11071 MB" red="22933 MB" 
    351: 2022-09-09 09:57:16 green="21726 MB" msg="Kernel enters memory conserve mode"
    352: 2022-09-09 09:57:17 MemTotal: 24719848 kB
    353: 2022-09-09 09:57:17 MemFree: 1235600 kB
    354: 2022-09-09 09:57:17 MemAvailable: 16023224 kB
    355: 2022-09-09 09:57:17 Buffers: 539112 kB
    356: 2022-09-09 09:57:17 Cached: 15736824 kB
    357: 2022-09-09 09:57:17 SwapCached: 0 kB
    358: 2022-09-09 09:57:17 Active: 16883340 kB
    359: 2022-09-09 09:57:17 Inactive: 1416028 kB
    360: 2022-09-09 09:57:17 Active(anon): 2869300 kB
    361: 2022-09-09 09:57:17 Inactive(anon): 489040 kB
    362: 2022-09-09 09:57:17 Active(file): 14014040 kB
    363: 2022-09-09 09:57:17 Inactive(file): 926988 kB


    443: 2022-09-13 09:21:16 service=kernel conserve=on total="24140 MB" used="9984 MB" red="22933 MB" 
    444: 2022-09-13 09:21:16 green="21726 MB" msg="Kernel enters memory conserve mode"
    445: 2022-09-13 09:21:16 MemTotal: 24719848 kB
    446: 2022-09-13 09:21:16 MemFree: 1235444 kB
    447: 2022-09-13 09:21:16 MemAvailable: 17261224 kB
    448: 2022-09-13 09:21:16 Buffers: 141636 kB
    449: 2022-09-13 09:21:16 Cached: 17210688 kB
    450: 2022-09-13 09:21:16 SwapCached: 0 kB
    451: 2022-09-13 09:21:16 Active: 16342428 kB
    452: 2022-09-13 09:21:16 Inactive: 2213932 kB
    453: 2022-09-13 09:21:16 Active(anon): 1766628 kB
    454: 2022-09-13 09:21:16 Inactive(anon): 476120 kB
    455: 2022-09-13 09:21:16 Active(file): 14575800 kB
    456: 2022-09-13 09:21:16 Inactive(file): 1737812 kB

     

    AEK
    srajeswaran
    Staff
    srajeswaranAnswer
    Staff
    October 8, 2023

    Can you confirm if the above conserve mode log examples are from 7.0.12 or an older version? On 7.0.7 and some other older versions due to bug - 798303, conserve mode was triggered even when the usage is lower than the red threshold .
    But this issue is fixed from 7.0.8 , so I don't expect you to see these logs on device running versions above 7.0.8 (or these logs should be generated before upgrade).

    Regarding the calculation of conserve mode, conserve mode memory calculation don't include the freeable memory.  The logs that you are shared are a buggy behavior, I hope this clarifies.

    Can you confirm if you have disk logging enabled? If so, you may need to disable it for now.

      Terms & ConditionsAccessibility statement