Technical Tip: Configuring and verifying FGSP with an IPsec tunnel and session synchronization

[sthampi_FTNT]

Description

This article provides configuration and verification steps to maintain an FGSP high availability cluster of SecureGateways.

Scope

FortiGate v7.2.3 GA.

Solution

Important Notes:

• In the diagram, FGT A and FGT B are geographically located at Site A while FGT C is at Site B.
• Layer 3 connectivity between FGT A, FGT B, and FGT C exists to synchronize IPSec SA and it's associated session information
• IPsec clients, typically the LTE endpoints, form an IPsec Tunnel with FGT A, B and C through the intermediary routers.
• Routers perform the task of load sharing the IPsec tunnels between different FGSP members.
• Traffic can be steered to the appropriate FortiGate without loss through the use of BGP Path attributes and features such as Local Preference, Conditional advertisements, Route maps, AS Path prepend and other metrics in conjunction with BFD. (This article will not explore this fine-tuning).
• If one of the FortiGates fails, Dynamic Routing and BFD will converge to find the next most suitable FGSP peer with minimum time delay.
• The configuration on the Fortigate needs to be consistent across all FGSP members to ensurea seamless failover with no disconnection of existing sessions.
• Note that the IPsec tunnel configuration, firewall policy, and addresses can be synchronized between one FGSP peer and another by configuring standalone-configuration-synchronization if L2 reachability is established.
• If L2 reachability is not present, the administrator must manually synchronize the configuration between the cluster members.

1. The following is the configuration on FGT_A and FGT_B for the IPsec VPN Gateway. Passive mode is enabled so that the Fortigate will not initiate the IPsec tunnel and will act as a responder. Uplink/downlink routers make the decision to send IPsec Tunnel traffic to the required FortiGate.

FGT-A # show vpn ipsec phase1-interface  

config vpn ipsec phase1-interface

edit "LTE_CLIENT"

set interface "Loopback_IP"

set ike-version 2

set peertype any

set net-device disable

set passive-mode enable

set proposal aes256-sha256

set fgsp-sync enable

set remote-gw 172.16.2.2

set psksecret ENC WOXsQ4ObCBwb6fAWWBlqxSV1xpQUJ..vHk7ru5gNA==

next

end

config vpn ipsec phase2-interface

edit "LTE_P2"

set phase1name "LTE_CLIENT"

set proposal aes256-sha256

next

end

2. The following is the configuration on FGT_A for BGP. FGT_B uses a similar configuration with a different IP address.
   

FGT-A # show router bgp

config router bgp

set as 500

set router-id 2.2.2.2

config neighbor

edit "10.100.3.119"

set remote-as 500

next

edit "10.103.3.120"

set remote-as 400

next

end

config network

edit 1

set prefix 172.16.1.1 255.255.255.255

next

end

config redistribute "static"

set status enable

set route-map "To_Server"

end

3. Configuration on Router A for BGP:
   

show router bgp

config router bgp

set as 500

set router-id 4.4.4.4

config neighbor

edit "10.100.3.109"

set next-hop-self enable

set remote-as 500

set connect-timer 1

set route-reflector-client enable

next

edit "10.103.3.118"

set next-hop-self enable

set soft-reconfiguration enable

set remote-as 500

set route-reflector-client enable

next

edit "10.106.3.121"

set soft-reconfiguration enable

set remote-as 1000

next

end

4. FGSP configuration on FGT_A:

FGT_A # show system standalone-cluster

config system standalone-cluster

set standalone-group-id 1

set group-member-id 1

set layer2-connection available

set session-sync-dev "port4"

config cluster-peer

edit 1

set peerip 10.0.20.118

set syncvd "root"

next

end

5. FGSP configuration on FGT_B

FGT_B # show sys standalone-cluster

config system standalone-cluster

set standalone-group-id 1

set group-member-id 2

set layer2-connection available

set session-sync-dev "port4"

config cluster-peer

edit 1

set peerip 10.0.20.109

set syncvd "root"

next

end
 

6. Verify whether FGSP is configured correctly with the following commands. Example outputs are provided.
   

FGT_A # diag sys ha standalone-peers
Group=1, ID=1
Detected-peers=1
Kernel standalone-peers: num=1.
peer0: vfid=0, peerip:port = 10.0.20.118:708, standalone_id=2
       session-type: send=1, recv=0
       packet-type: send=0, recv=0
Kernel standalone dev_base:
       standalone_id=0:
       standalone_id=1:

FGT_A # diag sys ha fgsp-zone
Local standalone-member-id: 1
FGSP peer_num = 1
peer[1]: standalone-member-id=2, IP=10.0.20.118, vd=root, prio=1

FGT_B # diag sys ha standalone-peers
Group=1, ID=2
Detected-peers=1
Kernel standalone-peers: num=1.
peer0: vfid=0, peerip:port = 10.0.20.109:708, standalone_id=1
       session-type: send=0, recv=1
       packet-type: send=0, recv=0
Kernel standalone dev_base:
       standalone_id=0:
       standalone_id=1:
       standalone_id=2:
       standalone_id=3:

FGT_B # diag sys ha fgsp-zone
Local standalone-member-id: 2
FGSP peer_num = 1
peer[1]: standalone-member-id=1, IP=10.0.20.109, vd=root, prio=1

7. Verify the FGSP status of IPsec tunnels on FGT_A and FGT_B. In the following case, FGT_A was actively processing traffic, which caused it to become L3 Primary. FGT_B is L3 Secondary. Note that, at the time of failover, the L3 HA Role will reverse when the uplink/downlink routers steer traffic towards FGT_B.

FGT_A # diag vpn ike gateway list

vd: root/0
name: LTE_CLIENT
version: 2
interface: 0
addr: 172.16.1.1:500 -> 172.16.2.2:500
tun_id: 172.16.2.2/::172.16.2.2
remote_location: 0.0.0.0
created: 3959s ago
L3-HA: primary mcid 1 traffic 0 last 42953474s ago ike 0 last 42953474s ago  hasync 0 last 42953474s ago route-clash 0
PPK: no
IKE SA: created 1/2 established 1/2 time 10/15/20 ms
IPsec SA: created 1/2 established 1/2 time 0/5/10 ms

 id/spi: 1 92010a0c754c19a4/b21721d96ade7bb6
 direction: responder
 status: established 3760-3760s ago = 10ms
 proposal: aes256-sha256
 child: no
 SK_ei: 96b5e851800c303f-3480ed951a54eab5-079c2b4520e04791-f298d7c1d2b39cd9
 SK_er: c64903e826925b98-395468d7bce57179-96d4c7731a678955-1374ad38ed490f55
 SK_ai: 5c250b372e182d37-fa41692117a7bad2-89ea9bd431f039df-5c03a1cf3243718a
 SK_ar: 40b8516de821b6ca-dd38b47ec75d627e-20b85128c5c1e6c6-9806878d4c532eb1
 PPK: no
 message-id sent/recv: 180/10
 lifetime/rekey: 86400/82369
 DPD sent/recv: 00000000/00000000

FGT_A # diag vpn tunnel list
list all ipsec tunnel in vd 0
------------------------------------------------------
name=LTE_CLIENT ver=2 serial=1 172.16.1.1:0->172.16.2.2:0 tun_id=172.16.2.2  tun_id6=::172.16.2.2 dst_mtu=1500 dpd-link=on weight=1
bound_if=0 lgwy=static/1 tun=intf/0 mode=auto/1 encap=none/520  options[0208]=npu frag-rfc run_state=0 role=sync-primary accept_traffic=1  overlay_id=0

proxyid_num=1 child_num=0 refcnt=4 ilast=2530 olast=2597 ad=/0
stat: rxp=73 txp=35 rxb=4013 txb=1860
dpd: mode=on-demand on=1 idle=20000ms retry=3 count=0 seqno=0
natt: mode=none draft=0 interval=0 remote_port=0
fec: egress=0 ingress=0
proxyid=LTE_P2 proto=0 sa=1 ref=3 serial=1
  src: 0:0.0.0.0/0.0.0.0:0
  dst: 0:0.0.0.0/0.0.0.0:0
  SA: ref=3 options=30202 type=00 soft=0 mtu=1438 expire=38558/0B      replaywin=2048
  seqno=24 esn=0 replaywin_lastseq=0000004a qat=0 rekey=0 hash_search_len=1
  life: type=01 bytes=0/0 timeout=42930/43200
  dec: spi=00f95461 esp=aes key=32      b45fdedcafa68530dc086d207a5f779d28bacac510d82dded2235246a5bd5d8c
  ah=sha256 key=32    e60289aad05caae9a7263430f386cb24645fc6066fd71ad51b103c0aa86dd28e
  enc: spi=8c94b745 esp=aes key=32  7b03e90445acf8223fe5c72b1467a3cf60fe54908967889c99b2c94d32d6ad6e
 ah=sha256 key=32  db84f3ebd99d3d428f402dd4701d652272395b2a3fa3e27d4c8365ee361a735d
 dec:pkts/bytes=146/8026, enc:pkts/bytes=70/6216
 npu_flag=00 npu_rgwy=172.16.2.2 npu_lgwy=172.16.1.1 npu_selid=0 dec_npuid=0 enc_npuid=0
 run_tally=0

FGT_B # diag vpn ike gateway list

vd: root/0
name: LTE_CLIENT
version: 2
interface: 0
addr: 172.16.1.1:500 -> 172.16.2.2:500
tun_id: 172.16.2.2/::172.16.2.2
remote_location: 0.0.0.0
created: 4770s ago
L3-HA: secondary mcid 1 traffic 0 last 42953519s ago ike 0 last 42953519s ago  hasync 0 last 42953519s ago route-clash 0
PPK: no
IKE SA: created 1/1 established 1/1 time 10/10/10 ms
IPsec SA: created 1/1 established 1/1 time 0/0/0 ms

 id/spi: 1 92010a0c754c19a4/b21721d96ade7bb6
 direction: responder
 status: established 4571-4571s ago = 10ms
 proposal: aes256-sha256
 child: no
 SK_ei: 96b5e851800c303f-3480ed951a54eab5-079c2b4520e04791-f298d7c1d2b39cd9
 SK_er: c64903e826925b98-395468d7bce57179-96d4c7731a678955-1374ad38ed490f55
 SK_ai: 5c250b372e182d37-fa41692117a7bad2-89ea9bd431f039df-5c03a1cf3243718a
 SK_ar: 40b8516de821b6ca-dd38b47ec75d627e-20b85128c5c1e6c6-9806878d4c532eb1
 PPK: no
 message-id sent/recv: 180/7
 lifetime/rekey: 86400/81558
 DPD sent/recv: 00000000/00000000

FGT_B # diag vpn tunnel list
list all ipsec tunnel in vd 0
------------------------------------------------------
name=LTE_CLIENT ver=2 serial=1 172.16.1.1:0->172.16.2.2:0 tun_id=172.16.2.2 tun_id6=::172.16.2.2 dst_mtu=0 dpd-link=on weight=1
bound_if=0 lgwy=static/1 tun=intf/0 mode=auto/1 encap=none/520 options[0208]=npu frag-rfc run_state=0 role=standby accept_traffic=1 overlay_id=0

proxyid_num=1 child_num=0 refcnt=4 ilast=42954878 olast=42954878 ad=/0
stat: rxp=0 txp=0 rxb=0 txb=0
dpd: mode=on-demand on=1 idle=20000ms retry=3 count=0 seqno=0
natt: mode=none draft=0 interval=0 remote_port=0
fec: egress=0 ingress=0
proxyid=LTE_P2 proto=0 sa=1 ref=2 serial=1
 src: 0:0.0.0.0/0.0.0.0:0
 dst: 0:0.0.0.0/0.0.0.0:0
 SA: ref=3 options=30202 type=00 soft=0 mtu=1280 expire=36998/0B replaywin=2048
  seqno=10000024 esn=0 replaywin_lastseq=0000004a qat=0 rekey=0    hash_search_len=1
 life: type=01 bytes=0/0 timeout=42480/42750
 dec: spi=00f95461 esp=aes key=32  b45fdedcafa68530dc086d207a5f779d28bacac510d82dded2235246a5bd5d8c
 ah=sha256 key=32  e60289aad05caae9a7263430f386cb24645fc6066fd71ad51b103c0aa86dd28e
 enc: spi=8c94b745 esp=aes key=32   7b03e90445acf8223fe5c72b1467a3cf60fe54908967889c99b2c94d32d6ad6e
 ah=sha256 key=32   db84f3ebd99d3d428f402dd4701d652272395b2a3fa3e27d4c8365ee361a735d
  dec:pkts/bytes=0/0, enc:pkts/bytes=0/0
  npu_flag=00 npu_rgwy=172.16.2.2 npu_lgwy=172.16.1.1 npu_selid=0 dec_npuid=0    enc_npuid=0
 run_tally=0

8. Run the following command to quickly verify how many tunnels are up or down on each FGSP member. An example output is attached.
   

get vpn ipsec stats tunnel 
tunnels
  total: 256
    static/ddns: 256
    dynamic: 0
    manual: 0
  errors: 45
selectors
  total: 256
  up: 251

9. Use the output of the following command to verify whether session statistics are also synchronized to the FGSP peers, particularly the synced flag on L3 HA primary and syn_ses on all L3 HA secondary nodes. If the session information is not present on the slave unit, user traffic may be dropped when failover occurs, requiring the user to reconnect.

FGT_A # diag sys session list

session info: proto=6 proto_state=01 duration=13 expire=3588 timeout=3600 flags=00000000 socktype=0 sockport=0 av_idx=0 use=3
origin-shaper=
reply-shaper=
per_ip_shaper=
class_id=0 ha_id=1:0 policy_dir=0 tunnel=/ vlan_cos=0/255
state=may_dirty npu synced
statistic(bytes/packets/allow_err): org=432/8/1 reply=372/7/1 tuples=2
tx speed(Bps/kbps): 33/0 rx speed(Bps/kbps): 28/0
orgin->sink: org pre->post, reply pre->post dev=19->5/5->19 gwy=10.103.3.120/172.16.2.2
hook=pre dir=org act=noop 10.108.3.123:40342->10.107.3.122:5005(0.0.0.0:0)
hook=post dir=reply act=noop 10.107.3.122:5005->10.108.3.123:40342(0.0.0.0:0)
pos/(before,after) 0/(0,0), 0/(0,0)
misc=0 policy_id=2 pol_uuid_idx=14729 auth_info=0 chk_client_info=0 vd=0
serial=000003e3 tos=ff/ff app_list=0 app=0 url_cat=0
rpdb_link_id=00000000 ngfwid=n/a
npu_state=0x000100
npu info: flag=0x00/0x82, offload=0/0, ips_offload=0/0, epid=0/0, ipid=0/0, vlan=0x0000/0x0000
vlifid=0/0, vtag_in=0x0000/0x0000 in_npu=0/0, out_npu=0/0, fwd_en=0/0, qid=0/0
no_ofld_reason:
total session 1

FGT_B # diag sys session list

session info: proto=6 proto_state=01 duration=70 expire=3529 timeout=3600 flags=00000000 socktype=0 sockport=0 av_idx=0 use=3
origin-shaper=
reply-shaper=
per_ip_shaper=
class_id=0 ha_id=1:0 policy_dir=0 tunnel=/ vlan_cos=0/255
state=may_dirty syn_ses
statistic(bytes/packets/allow_err): org=0/0/0 reply=0/0/0 tuples=2
tx speed(Bps/kbps): 0/0 rx speed(Bps/kbps): 0/0
orgin->sink: org pre->post, reply pre->post dev=19->5/5->19 gwy=0.0.0.0/0.0.0.0
hook=pre dir=org act=noop 10.108.3.123:40342->10.107.3.122:5005(0.0.0.0:0)
hook=post dir=reply act=noop 10.107.3.122:5005->10.108.3.123:40342(0.0.0.0:0)
pos/(before,after) 0/(0,0), 0/(0,0)
misc=0 policy_id=2 pol_uuid_idx=0 auth_info=0 chk_client_info=0 vd=0
serial=000003e3 tos=ff/ff app_list=0 app=0 url_cat=0
rpdb_link_id=00000000 ngfwid=n/a
npu_state=0x000100
no_ofld_reason: npu-flag-off
total session 1