Technical Tip: Configuring link redundancy, Traffic load-balancing/load-sharing, ECMP (Equal Cost Multiple Path), Dual Internet or WAN scenario

Description

This article describes how to configure load-balancing over multiple interfaces (multiple ISPs - dual [or more] WAN connections, for example) and implement the link redundancy (fail-over).

'ECMP' stands for 'Equal Cost Multiple Path'. ECMP is a mechanism that allows multiple routes to the same destination with different next-hops and load-balances traffic over those multiple next-hops.

ECMP implementation on the FortiGate:

•    ECMP is supported for:

• Static Routing.
• OSPF.
• BGP.
  

•    ECMP only works for routes that are sourced by the same routing protocol (i.e, Static Route, OSPF, or BGP).
•    ECMP is enabled by default with 10 paths.
•    ECMP with static routes is effective if the routes are configured with the same distance and the same priority.

ECMP Distribution algorithm:

There are three configuration options for ECMP route failover and load balancing:

• Source-based (also called source IP-based - default setting);
  
• Weighted (also called weight-based);
  
• Spill-over (also called usage-based).

• Each new source device (for example, the PC in the diagram) crossing the FortiGate will use one of the 3 paths to the Internet,
• All traffic originating from the same source IP is expected to 'always' use the same path.

Scope

Solution

The diagram below can be used to illustrate this article: the FortiGate has 3 different interfaces (physical or VLANs) to reach the Internet, and it is desired to use all 3 of them to load-balance traffic and redundancy.

Or in a dual WAN scenario:


Or over the same interface with different next-hops:

Expectations, requirements:
Firewall policies should be set for each path to allow traffic to flow on each Internet port.

Configuration:
Note: ECMP is a per-VDOM setting (from CLI only).

config system settings
(settings) # set ecmp-max-paths    (10 is default)
end

Configuration example: Static routes defaulting to the Internet

This is the CLI example to configure 3 different routes to the same destination (in this case, they will be default routes). Note that in this example, the FortiGate unit will use the default source-based distribution algorithm.

config router static
    edit 1
        set device "port1"
        set gateway 192.168.2.2
    next

   edit 2
        set device "port2"
        set gateway 192.168.3.2
    next

   edit 3
        set device "port3"
        set gateway 192.168.4.2
    next
end

Configuration example: BGP ECMP settings.

This is the CLI example to configure BGP different routes to the same destination (in this case, they will NOT be the default routes). Note that in this example, the FortiGate unit will use the default source-based distribution algorithm.

config system settings
(settings) # set ecmp-max-paths 10
(settings) # set v4-ecmp-mode source-ip-based
end

In the BGP configuration, enable one or both of the following settings:

config router bgp
    set ebgp-multipath enable <----- ECMP will be selected for EBGP routes.
    set ibgp-multipath enable <----- ECMP will be selected for IBGP routes.
end

config router bgp
    set as 65001
    set router-id 172.31.19.186
    set holdtime-timer 30
    set ebgp-multipath enable
        config neighbor
            edit "172.16.18.195"
                set remote-as 64516
                set weight 10
            next
            edit "172.16.19.240"
                set remote-as 64516
                set weight 10 <<--- Not used in ECMP, configure set v4-ecmp-mode weight-based in 'config system settings'
            next
        end

Notes: 

• BGP routes do not support the weight setting for ECMP weight-based LLB, so it is necessary to set v4-ecmp-mode weight-based in 'config system settings'. BGP will continue with round-robin behavior if configured with an ECMP weight-based algorithm.
• If deterministic-med is enabled, BGP ECMP will be bypassed and the routes from the same AS are grouped. The best routes of each group will be compared. While selecting the best route within one group, no ECMP is considered; there is only one best route for one group.

config router bgp
    set always-compare-med enable
    set deterministic-med enable
end

The following conditions are considered to select the best route for a group:

1. Weight check.
2. Local preference check.
   
3. Local route check.
   
4. AS path length check.
   
5. Origin check.
   
6. MED check.
   
7. Peer type check.
   
8. IGP metric check.
   

Related documents:
Advanced static routing example: ECMP failover and load balancing
Multipath Routing Basics
Client-Side SD-WAN with IPsec VPN Deployment Scenario – Expert

Verification.
Static Routes:

Check the routing table of the FortiGate unit and look for the 3 routes configured :

FGT1 # get router info routing-table all

S*      0.0.0.0/0 [10/0] via 192.168.2.2, port1
                          [10/0] via 192.168.3.2, port2
                          [10/0] via 192.168.4.2, port3
C       192.168.1.0/24 is directly connected, internal
C       192.168.2.0/24 is directly connected, port1
C       192.168.3.0/24 is directly connected, port2
C       192.168.4.0/24 is directly connected, port3

Note: If the FortiGate unit was configured with different next-hops over the same interface, the routing table would be:

FGT # get router info routing-table all

S    *> 0.0.0.0/0 [10/0] via 172.16.224.223, port2
     *>           [10/0] via 172.16.224.224, port2
C    *> 172.16.224.0/23 is directly connected, port2

BGP Routes: Check the routing table of the FortiGate unit and look for the BGP routes:

FGT1 # get router info routing-table all
Codes: K - kernel, C - connected, S - static, R - RIP, B - BGP
       O - OSPF, IA - OSPF inter area
       N1 - OSPF NSSA external type 1, N2 - OSPF NSSA external type 2
       E1 - OSPF external type 1, E2 - OSPF external type 2
       i - IS-IS, L1 - IS-IS level-1, L2 - IS-IS level-2, ia - IS-IS inter area
       * - candidate default

S* 0.0.0.0/0 [10/0] via 172.16.19.254, port2
B 10.58.0.0/22 [20/0] via 172.16.18.195, port2, 00:59:09
               [20/0] via 172.16.19.240, port2, 00:59:09
C 10.129.0.0/22 is directly connected, port1
C 172.16.16.0/22 is directly connected, port2

FGT # get router info bgp network 10.58.0.0/22
BGP routing table entry for 10.58.0.0/22
Paths: (2 available, best 1, table Default-IP-Routing-Table)
  Advertised to non peer-group peers:
   172.16.19.240
  64516
    172.16.18.195 from 172.16.18.195 (172.16.18.195)
      Origin IGP metric 0, localpref 100, weight 10, valid, external, best
      Last update: Mon Sep 28 08:10:10 2015

  64516
    172.16.19.240 from 172.16.19.240 (172.16.19.240)
      Origin IGP metric 0, localpref 100, weight 10, valid, external
      Last update: Mon Sep 28 08:10:10 2015

Troubleshooting:

Identifying which outgoing interface is used when ECMP is enabled can be done easily using the session table (policy id).

diagnose sniffer packet any '<filter>' 4

Related articles:

Technical Tip: Configuring Dual Internet Links (Design Considerations)

Technical Note: Detecting a link failure using Dead Gateway Detection (ping server) to ensure a link fail-over

Troubleshooting Tip: FortiOS routing (RIP, OSPF, BGP, static routes, ECMP)

Technical Tip: FortiGate routing table conditions

Technical Tip: Routing behavior depending on distance and priority for static routes, and Policy Based Routes

Technical Note : Identical next hops in the routing table, over different FortiGate interfaces