|
Why Long-Term Focus is Needed:
Conserve mode on FortiGate devices can be triggered by various factors, including traffic patterns, functions, policies, and features. However, three primary categories contribute to conserve mode:
- Insufficient Memory Capacity: The combination of traffic patterns, functions, policies, and features may be too much for the device to handle, indicating a need for a firewall with a larger memory capacity.
- Unintended software behaviors: Memory leaks or functions that consume more memory than intended can cause conserve mode.
- Memory-Intensive Processes: Certain processes, such as FortiGuard updates, can consume additional memory, potentially surpassing the conserve mode threshold.
Understanding Memory Leaks:
A memory leak occurs when a program allocates memory but fails to release the memory back to the system when the process no longer needs the memory. This can happen due to:
- Unintentional memory allocation.
- Circular references.
- Unclosed resources.
Strategy for Memory Management:
To prevent the conserve mode, the administrator needs to understand the memory consumption and trends over time.
This involves:
- Measuring Memory Consumption: Measure memory at the lowest point (low traffic) and at the highest point (most users online).
- Identifying Specific Processes: Identify specific processes that may be consuming more memory.
Automating Monitoring: Monitoring this over the long term (months) is the most important element.
- Automation: Automate monitoring using scripts, such as Terra Term scripting or FortiGate's scripting tool. There are parameters that need to be obtained multiple times per day.
Key Parameters to Review:
Memory Consumption: Review memory consumption using the command:
F4 # get system performance status | grep Memory
Memory: 2040712k total, 1282352k used (62.8%), 439432k free (21.5%), 318928k freeable (15.7%)
- Memory after Reboot: How much memory is consumed immediately after a reboot.
- Memory at Low Traffic: How much memory is consumed during periods of low traffic.
- Memory at High Traffic: How much memory is consumed during periods of high traffic.
- Trend over Time: What is the trend of memory consumption over time? Is the memory increasing, decreasing, or stable.
- Session Information: Review session information using the command
F4 # get system performance status | grep sessions
Average sessions: 15 sessions in 1 minute, 13 sessions in 10 minutes, 13 sessions in 30 minutes
Maximal sessions: 18 sessions in 1 minute, 19 sessions in 10 minutes, 21 sessions in 30 minutes
Average session setup rate: 0 sessions per second in last 1 minute, 0 sessions per second in last 10 minutes, 0 sessions per second in last 30 minutes
Maximal session setup rate: 2 sessions per second in last 1 minute, 2 sessions per second in last 10 minutes, 2 sessions per second in last 30 minutes
Average NPU sessions: 0 sessions in last 1 minute, 0 sessions in last 10 minutes, 0 sessions in last 30 minutes
Maximal NPU sessions: 0 sessions in last 1 minute, 0 sessions in last 10 minutes, 0 sessions in last 30 minutes
- High Memory and Sessions: Is high memory consumption increasing with the number of sessions.
- Low Memory and Sessions: Is memory consumption decreasing as the number of sessions reduces.
- Conserve Mode Threshold: At any point, is the memory consumption near the conserve mode threshold (65% or more). Especially at night or a few days after a reboot.
- Process Memory Consumption: Review process memory consumption using the command: diag sys top-mem 20
F4 # diag sys top-mem 20
wad (2132): 106106kB
node (2013): 99512kB
ipshelper (2037): 33463kB
- Time: How does memory consumption change over time for specific processes.
- Traffic Volume: How does memory consumption change with traffic volume for specific processes.
- Evidence of Memory Release: Is there evidence of memory release for specific processes.
Preventive Actions:
- Capacity Planning: Ensures FortiGate devices handle current and future demands.
- Optimize configuration: Reduce memory consumption by effectively transforming design and configuration of security policies to ensure lower memory usage.
- Short-term preventive actions: Using the information obtained with the strategy above, some actions can be taken. Details below in distinct 3 scenarios.
Scenarios:
The following scenario describes how to read, interpret, and build an action plan
Scenario 1: High Memory Consumption during High Traffic
- Symptoms: Firewall just rebooted, memory utilization during non-production hours is 40% while in production is 67%
- Analysis: The firewall is operating near capacity, any traffic pattern change such as coordinated software updates in multiple endpoints, might trigger conserve mode.
- Action: Optimize memory consumption by disabling unused features, reducing UTM scanning, fine-tuning IPS and AV engines and adjusting logging and caching settings.
Scenario 2: Memory consumption increases over time.
- Symptoms: Firewall just rebooted, memory utilization during non-production hours is 31% while in production is 37%. Memory consumption steadily climbs. After 3 months, memory consumption is 51% and 68% respectively. A specific process named WAD was identified as a major process its memory consumption changed over time.
- Analysis: The firewall shows indications of memory leak which is a behavior that is not expected. Same as scenario 2, changes could trigger conserve mode after weeks or months after the last reboot or process restart.
- Action: Short term, restart the WAD process. Long-term plan, review findings with TAC. the command used 'diag sys kill 9 <pid>
Scenario 3: Memory Spikes During FortiGuard Updates or Configuration Changes
- Symptoms: Memory usage remains low (e.g., 40%) under normal conditions but temporarily spikes to 65%. Memory trend over time remains steady with temporary peaks detected, but no process is identified.
- Action: Changed the frequency of memory polling scripts to detect these spikes.
Conclusion:
Effective memory management is crucial to prevent the conserve mode on FortiGate devices. By understanding baseline memory consumption, identifying specific processes, and automating monitoring, the user can take proactive steps to ensure optimal performance and prevent conserve mode.
Related documents:
Troubleshooting Tip: Conserve mode due to IPS Engine or WAD
Troubleshooting Tip: High memory and High CPU general script using Tera Term
Technical Tip: FortiGate is entering into Conserve Mode during FortiGuard Updates
Technical Tip: How to stop and restart the IPS engine, verify status
Technical Tip: How to optimize the Memory consumption
Technical Tip: Use a new FortiOS mechanism to automatically restart WAD workers
|
