When the 'local-as' and 'local-as-replace-as' settings are configured in the config neighbor section of the BGP configuration on a FortiGate device, a network loop condition may occur. This happens if updates that the FortiGate initially generated are received back from another BGP neighbor.
As a result, the BGPD daemon may use up to 99% of the CPU resources and significantly more memory (around 50%), potentially causing the system to enter memory conserve mode.
The issue is expected to be resolved in FortiOS versions v7.2.10, v7.4.5, and v7.6.1, tentatively. Refer to the product release notes to confirm the fix once the firmware is officially released and available for download from the Support Portal.
Workaround:
An inbound route-map must be implemented to block incoming updates containing the same AS as the Local FortiGate. Use the following regex string in the route-map: '.*Local_AS.*Local_AS.*'. This expression matches any BGP prefix/update that includes the local AS at least twice.
- Configure an AS Path list.
config router aspath-list
edit "AS30"
config rule
edit 1
set action permit set regexp ".*30.*30.*"
next
end
next
end
config router route-map
edit "INBOUND"
config rule
edit 1
set action deny set match-as-path "AS30"
next
edit 2
next
end
next
end
- Apply the route-map at the Inbound direction.
config router bgp
set as 100 set router-id 1.1.1.1 set ebgp-multipath enable set ibgp-multipath enable set network-import-check disable set additional-path enable set scan-time 10 set graceful-restart enable set additional-path-select 255
config neighbor
edit "10.203.1.164"
set advertisement-interval 1 set soft-reconfiguration enable set remote-as 40 set local-as 30 set local-as-replace-as enable set route-map-in "INBOUND"
next
end
end
Symptoms:
- The FortiGate system will enter into conserve mode when the memory usage is 88% or above.
get system performance status Memory: 20583060k total, 18779868k used (91.2%), 1323960k free (6.4%), 479232k freeable (2.4%)
- The BGPD process consumes more than a normal amount of memory.
diagnose sys top 2 99 1 Run Time: 0 days, 9 hours and 58 minutes 2U, 0N, 7S, 9I, 79WA, 0HI, 3SI, 0ST; 20100T, 183F bgpd 3912 R < 99.0 57.1 15
- The network prefixes learned by the local FortiGate will display an increasingly long list of AS numbers in the output of the get router info bgp network. This list continues to grow until the network loop is prevented.
VRF 0 BGP table version is 1, local router ID is 192.168.168.253 Status codes: s suppressed, d damped, h history, * valid, > best, i - internal, S Stale Origin codes: i - IGP, e - EGP, ? - incomplete Network Next Hop Metric LocPrf Weight RouteTag Path *> 100.100.100.100/32 10.204.3.164 0 0 0 30 40 40 30 30 40 40 30 30 40 40 30 30 40 40 30 30 40 40 30 30 40 40 30 30 40 40 30 30 40 40 30 30 40 40 30 30 40 40 30 30 40 40 30 30 40 40 30 30 40 40 30 30 40 40 30 30 40 40 30 30 40 40 30 30 40 40 30 30 40 40 30 30 40 40 30 30 40 40 30 30 40 40 30 30 40 40 30 30 40 40 30 30 40 40 30 30 40 40 30 30 40 40 30 i <-/1>
Data collection plan:
To report any new issues related to memory consumption by the BGPD process, collect the following debug data during both normal and abnormal memory consumption periods of the daemon before submitting a support request to the Fortinet Technical Team.
Note:
- For collecting data during a normal memory consumption period, a restart of the relevant process or a reboot of the FortiGate device may be required which should be scheduled as part of a maintenance activity.
- Post daemon restart or FortiGate reboot, another iteration of the following debug data must be captured for baseline and comparison purposes.
- Depending on the user process that is restarted, end users may experience traffic outage.
- Identify the process ID of the BGP daemon to capture process dumps and traces:
diagnose sys process pidof bgpd diag sys process trace <PID of bgpd> diag sys process dump <PID of bgpd> diag sys process pstack <PID of bgpd> diag sys process sock-mem <PID of bgpd>
- Enable BGP debugging:
diagnose debug reset diagnose ip router bgp all enable diagnose ip router bgp level info diagnose debug console timestamp enable diagnose debug duration 2 diagnose debug enable .
The debug will stop printing data after 2 minutes. To permanently disable/reset the debug commands, execute the following commands:
diagnose debug disable diagnose debug reset
- Capture the output of the following commands:
execute tac report get router info bgp network get router info bgp memory get router info bgp summary get router info bgp neighbors
- Export the Router Events and System Event Logs from FortiGate.
|