Technical Tip : Configuring link redundancy - Traffic load-balancing / load-sharing - ECMP (Equal Cost Multiple Path) - Dual Internet or WAN scenario

[rmetzger]

Purpose
This article describes how to configure load-balancing over multiple interfaces (multiple ISPs - dual [or more] WAN connections, for example) and implement the link redundancy (fail-over).

'ECMP' stands for 'Equal Cost Multiple Path'. ECMP is a mechanism that allows multiple routes to the same destination with different next-hops and load-balances routed traffic over those multiple next-hops.

ECMP implementation on the FortiGate:

•    ECMP is supported for

    - Static Routing    
    - OSPF
    - BGP

•    ECMP only works for routes that are sourced by the same routing protocol (i.e: Static Route, OSPF or BGP).
•    ECMP is enabled by default with 10 paths.
•    ECMP with static routes is effective if the routes are configured with the same distance and same priority.

ECMP Distribution algorithm:

There are three configuration options for ECMP route failover and load balancing:

- Source based (also called source IP based - default setting );
- Weighted (also called weight-based);
- Spill-over (also called usage-based).

See more detail about those 3 modes in the technical documentation here.

Note about Source based:

As ECMP Fortinet algorithm is IP source hash based on a pre-NAT'ed IP address: 
 - each new source device (for example, the PC in the diagram) crossing the FortiGate will use one of the 3 paths to the Internet,

 - all traffic originating from the same source IP is expected to *always* use the same path.

FAIL OVER: Should one of the interface fail and be removed from the routing table, the traffic will be routed over the remaining routes. In the example describer later, no specific configuration is necessary for the route fail-over.

Scope

FortiGate models running FortiOS firmware versions 4.x, 5.x

FortiGate unit or VDOM in NAT mode only

Diagram

The diagram below can be used to illustrate this article: the FortiGate has 3 different interfaces (physical or VLANs) to reach the Internet, and we want to use all 3 of them to load-balance traffic and redundancy.

       [            ] port1 ---- [ Internet ]
LAN ===[ FortiGate  ] port2 ---- [ Internet ]
       [            ] port3 ---- [ Internet ]

or in a dual WAN scenario:


       [            ] wan1---- [ Internet ]
LAN ===[ FortiGate  ] wan2---- [ Internet ]


or over the same interface with different next-hops:

       [            ] wan1--[l2 switch]-- [ router1]
LAN ===[ FortiGate  ] wan1--[l2 switch]-- [ router2]

Expectations, Requirements
Firewall policies should be set for each path to allow traffic to flow on each Internet ports.

Configuration
Note: ECMP is a per-VDOM setting (from CLI only).

# config system settings
    (settings) # set ecmp-max-paths    (10 is default)
end

Configuration example: Static routes defaulting to the Internet

This is the CLI example to configure 3 different routes to the same destination (in this case, they will be default routes). Note, that in this example the FortiGate unit will use the default source-based distribution algorithm.

# config router static
    edit 1
        set device "port1"
        set gateway 192.168.2.2
    next

   edit 2
        set device "port2"
        set gateway 192.168.3.2
    next

   edit 3
        set device "port3"
        set gateway 192.168.4.2
    next
end

Configuration example: BGP ECMP settings

This is the CLI example to configure BGP different routes to the same destination (in this case, they will NOT be the default routes). Note, that in this example the FortiGate unit will use the default source-based distribution algorithm.

# config system settings
    (settings) # set ecmp-max-paths 10
    (settings) # set v4-ecmp-mode source-ip-based
end

In the BGP configuration  enable one or both of the following settings:

# config router bgp
    set ebgp-multipath enable <<<<----- ECMP will be selected for EBGP routes.
    set ibgp-multipath enable <<<<----- ECMP will be selected for IBGP routes.
end

 # config router bgp
    set as 65001
    set router-id 172.31.19.186
    set holdtime-timer 30
    set ebgp-multipath enable
        config neighbor
            edit "172.16.18.195"
                set remote-as 64516
                set weight 10
            next
            edit "172.16.19.240"
                set remote-as 64516
                set weight 10 <<--- Not used in ECMP, configure set v4-ecmp-mode weight-based in 'config system settings'
            next
        end

Note.

BGP routes do not support the weight setting for ECMP weight based LLB so need to set v4-ecmp-mode weight-based in 'config system settings'. BGP will continue with round-robin behavior if configured with ECMP weight-based algorithm.

Note.

 If deterministic-med is enabled, BGP ECMP will be bypassed and the routes from the same AS are grouped together. The best routes of each group will be compared. While selecting the best route within one group, no ECMP is considered, there is only one best route for one group.

# config router bgp
    set always-compare-med enable
    set deterministic-med enable
end

The following conditions are considered to select the best route for a group:

1. Weight check
2. Local preference check
3. Local route check
4. AS path length check
5. Origin check
6. MED check
7. Peer type check
8. IGP metric check

Links:
Advanced static routing example: ECMP failover and load balancing
Multipath Routing Basics
Client-Side SD-WAN with IPsec VPN Deployment Scenario – Expert

Verification
Static Routes:

Check the routing table of the FortiGate unit and look for the 3 routes configured :

# FGT1 # get router info routing-table all

S*      0.0.0.0/0 [10/0] via 192.168.2.2, port1                           [10/0] via 192.168.3.2, port2                           [10/0] via 192.168.4.2, port3 C       192.168.1.0/24 is directly connected, internal C       192.168.2.0/24 is directly connected, port1 C       192.168.3.0/24 is directly connected, port2 C       192.168.4.0/24 is directly connected, port3

Note.

If the FortiGate unit was configured with different next-hops over the same interface, the routing table would be:

# FGT # get router info routing-table all

S    *> 0.0.0.0/0 [10/0] via 172.16.224.223, port2
     *>           [10/0] via 172.16.224.224, port2
C    *> 172.16.224.0/23 is directly connected, port2

BGP Routes:

Check the routing table of the FortiGate unit and look for the BGP routes:

# FGT1 # get router info routing-table all
Codes: K - kernel, C - connected, S - static, R - RIP, B - BGP
       O - OSPF, IA - OSPF inter area
       N1 - OSPF NSSA external type 1, N2 - OSPF NSSA external type 2
       E1 - OSPF external type 1, E2 - OSPF external type 2
       i - IS-IS, L1 - IS-IS level-1, L2 - IS-IS level-2, ia - IS-IS inter area
       * - candidate default

S* 0.0.0.0/0 [10/0] via 172.16.19.254, port2
B 10.58.0.0/22 [20/0] via 172.16.18.195, port2, 00:59:09
               [20/0] via 172.16.19.240, port2, 00:59:09
C 10.129.0.0/22 is directly connected, port1
C 172.16.16.0/22 is directly connected, port2

# FGT # get router info bgp network 10.58.0.0/22
BGP routing table entry for 10.58.0.0/22
Paths: (2 available, best 1, table Default-IP-Routing-Table)
  Advertised to non peer-group peers:
   172.16.19.240
  64516
    172.16.18.195 from 172.16.18.195 (172.16.18.195)
      Origin IGP metric 0, localpref 100, weight 10, valid, external, best
      Last update: Mon Sep 28 08:10:10 2015

  64516
    172.16.19.240 from 172.16.19.240 (172.16.19.240)
      Origin IGP metric 0, localpref 100, weight 10, valid, external
      Last update: Mon Sep 28 08:10:10 2015

Troubleshooting

Identifying what outgoing interface is used when ECMP is enabled can be done easily using the session table (policy id).

 
From the GUI, Go to System -> Status, identify the session and check the policy ID.

Refer to the policy ID in the Firewall table to find out which interface is used.

Using the FortiGate sniffer using interface 'any' and level 4 would show the egressing interface used.

For example:

# diagnose sniffer packet any '<filter>' 4

Related Articles

Configuring a Default Route (Default Gateway) on a FortiGate in NAT mode - REMOVED from public KB

Configuring Dual Internet Links (Design Considerations)

Technical Note: Detecting a link failure using Dead Gateway Detection (ping server) to ensure a link...

Troubleshooting tips for FortiOS routing (RIP, OSPF, BGP, static routes, ECMP)

Technical Tip: FortiGate routing table conditions

Technical Note: Routing behavior depending on distance and priority for static routes, and Policy Ba...

Technical Note : Identical next hops in the routing table, over different FortiGate interfaces