Technical Tip: Virtual Domain ID consideration when Troubleshooting HA Clusters

[ChrisSuri]

Description	This article describes issues that can arise when running diagnostic commands across FortiGate Cluster members where Virtual Domains are configured.
Scope	FortiGate Cluster with Virtual Domains Configured.
Solution	Virtual Domains are assigned a numerical ID as well as an Alphanumeric Name. It could be assumed that the Virtual Domain numerical ID when viewed across cluster members would be the same, however, this is not always the case. As a consequence, when using diagnostic commands that can accept a Virtual Domain numerical ID or Virtual Domain Name, it is advisable to validate the numerical ID or to utilize the Virtual Domain Name. The below example shows diagnostic command outputs from a FortiGate cluster with two members, FW1 and FW2. There are several Virtual Domains defined, but we focus on the CORE1 Virtual Domain. As we can see using the command 'diagnose system vd list' the Numerical ID differs across the Firewalls, but the Alphanumeric Name is consistent. FW1 (global) # diagnose sys vd list | grep CORE1 name=CORE1/CORE1 index=1 enabled fib_ver=35 rpdb_ver=0 use=348 rt_num=62 asym_rt=0 sip_helper=0, sip_nat_trace=1, mc_fwd=0, mc_ttl_nc=0, tpmc_sk_pl=0 FW2 (global) # diagnose sys vd list | grep CORE1 name=CORE1/CORE1 index=2 enabled fib_ver=148 rpdb_ver=0 use=297 rt_num=12 asym_rt=0 sip_helper=0, sip_nat_trace=1, mc_fwd=0, mc_ttl_nc=0, tpmc_sk_pl=0 This behavior is important to take into account when utilizing commands such as 'diag sys session list' as it is possible to pass a filter that relies on the Numerical ID, or the Alphanumeric name, using the latter can result in incorrect conclusions that a session for example is not synchronized across nodes. In the following example, there is an SSH session traversing the CORE1 virtual domain, terminating on the Server with IP address 192.168.251.254 on port 22. The administrator has already checked the VDOM ID on FW1, which is '1' and uses the same commands on each cluster member to filter sessions using the following filter settings: FW1 (global) # diagnose sys session filter session filter: vd: 1 sintf: any dintf: any proto: any proto-state: any source ip: any NAT'd source ip: any dest ip: 192.168.251.254-192.168.251.254 source port: any NAT'd source port: any dest port: 22-22 policy id: any expire: any duration: any state1: any state2: any When the command 'diag sys session list' is run on FW1, the expected session is visible: FW1 (global) # diagnose sys session list session info: proto=6 proto_state=11 duration=11 expire=3588 timeout=3600 flags=00000000 socktype=0 sockport=0 av_idx=0 use=5 origin-shaper= reply-shaper= per_ip_shaper= class_id=0 ha_id=0 policy_dir=0 tunnel=/ vlan_cos=0/255 state=log may_dirty ndr npu synced f00 app_valid log-start statistic(bytes/packets/allow_err): org=92/2/1 reply=52/1/1 tuples=3 tx speed(Bps/kbps): 0/0 rx speed(Bps/kbps): 0/0 orgin->sink: org pre->post, reply pre->post dev=49->50/50->49 gwy=192.168.251.254/192.168.250.2 hook=post dir=org act=snat 192.168.250.2:49346->192.168.251.254:22(192.168.251.1:49346) hook=pre dir=reply act=dnat 192.168.251.254:22->192.168.251.1:49346(192.168.250.2:49346) hook=post dir=reply act=noop 192.168.251.254:22->192.168.250.2:49346(0.0.0.0:0) pos/(before,after) 0/(0,0), 0/(0,0) src_mac=00:61:6c:70:11:01 dst_mac=00:61:6c:70:0d:03 misc=0 policy_id=1 pol_uuid_idx=589 auth_info=0 chk_client_info=0 vd=1 serial=000009b2 tos=ff/ff app_list=6000 app=16060 url_cat=0 rpdb_link_id=00000000 ngfwid=n/a npu_state=0x003c94 ips_offload ofld-O ofld-R npu info: flag=0x81/0x81, offload=8/8, ips_offload=1/1, epid=16/16, ipid=70/70, vlan=0x00fa/0x00fb vlifid=70/70, vtag_in=0x00fa/0x00fb in_npu=1/1, out_npu=1/1, fwd_en=0/0, qid=3/5 total session 1 However, when setting the same filters, and run the command 'diagnose sys session list' on FW2, no sessions are visible: FW2 (global) # diagnose sys session filter session filter: vd: 1 sintf: any dintf: any proto: any proto-state: any source ip: any NAT'd source ip: any dest ip: 192.168.251.254-192.168.251.254 source port: any NAT'd source port: any dest port: 22-22 policy id: any expire: any duration: any state1: any state2: any FW2 (global) # diagnose sys session list total session 0 The reason for this is that the assumption was made initially that the Virtual Domain numerical ID is the same across the cluster members, but that is not true in this case. If adjusting the session filter on FW2 to reference the Virtual Domain ID '2', the session is present: FW2 (global) # diagnose sys session filter vd 2 FW2 (global) # diagnose sys session list session info: proto=6 proto_state=11 duration=221 expire=3378 timeout=3600 flags=00000000 socktype=0 sockport=0 av_idx=0 use=4 origin-shaper= reply-shaper= per_ip_shaper= class_id=0 ha_id=0 policy_dir=0 tunnel=/ vlan_cos=0/255 state=log dirty may_dirty ndr npu f00 syn_ses app_valid statistic(bytes/packets/allow_err): org=0/0/0 reply=0/0/0 tuples=3 tx speed(Bps/kbps): 0/0 rx speed(Bps/kbps): 0/0 orgin->sink: org pre->post, reply pre->post dev=49->50/50->49 gwy=0.0.0.0/0.0.0.0 hook=post dir=org act=snat 192.168.250.2:49346->192.168.251.254:22(192.168.251.1:49346) hook=pre dir=reply act=dnat 192.168.251.254:22->192.168.251.1:49346(192.168.250.2:49346) hook=post dir=reply act=noop 192.168.251.254:22->192.168.250.2:49346(0.0.0.0:0) pos/(before,after) 0/(0,0), 0/(0,0) misc=0 policy_id=1 pol_uuid_idx=0 auth_info=0 chk_client_info=0 vd=2 serial=000009b2 tos=ff/ff app_list=6000 app=16060 url_cat=0 rpdb_link_id=00000000 ngfwid=n/a npu_state=0x001000 npu info: flag=0x00/0x00, offload=0/0, ips_offload=0/0, epid=0/0, ipid=0/0, vlan=0x0000/0x0000 vlifid=0/0, vtag_in=0x0000/0x0000 in_npu=0/0, out_npu=0/0, fwd_en=0/0, qid=0/0 no_ofld_reason: redir-to-ips total session 1 If the Virtual Domain Name is used as opposed to the Virtual Domain ID,  it is possible to see that when the session filter is set, the firewall resolves the Virtual Domain name to the correct Virtual Domain Numerical ID: FW1 (global) # diagnose sys session filter vd-name CORE1 find VDOM ID: 1 FW2 (global) # diagnose sys session filter vd-name CORE1 find VDOM ID: 2 Care should be taken when working with Virtual Domains, and assuming that their numerical ID is consistent across cluster members, as demonstrated in this example. This is not always the case. Particular caution should be exercised when utilizing destructive commands e.g. 'diag sys session clear', if the wrong Virtual Domain Numerical ID is used across cluster members, clear the wrong Virtual Domain session table. Related Articles Technical Tip: How it is possible to use 'filter' with 'diagnose sys session list' command to get th... Technical Tip: Using filters to clear sessions on a FortiGate Troubleshooting Tip: FortiGate session table information