Connectivity Issue Between 2 Fortigate ?!

[huud]

Hi,

I have 2 Fortigate setup as below, and the issue as follows.

FW1 cannot ping anything on FW2. network, but FW2 can ping FW1's 10.11.30 and 10.11.40 network.

BGP is setup between the 2 and is working fine, with RouteMap and Prefix List, so traffic from 10.11.30.0 flows through port1 (192.168.9.181) and 10.11.40.0 through port2 (192.168.9.182), smae configuration is on FW2 for 10.21 network.

FW1

Routing table for VRF=0
C 10.11.30.0/24 is directly connected, VLAN30
C 10.11.40.0/24 is directly connected, VLAN40
B 10.21.30.0/24 [20/0] via 192.168.9.182 (recursive is directly connected, port1), 00:18:07, [1/0]
B 10.21.40.0/24 [20/0] via 192.168.10.182 (recursive is directly connected, port2), 00:17:40, [1/0]
C 192.168.9.0/24 is directly connected, port1
C 192.168.10.0/24 is directly connected, port2

FW2

Routing table for VRF=0
B 10.11.30.0/24 [20/0] via 192.168.9.181 (recursive is directly connected, port1), 00:18:47, [1/0]
B 10.11.40.0/24 [20/0] via 192.168.10.181 (recursive is directly connected, port2), 00:18:15, [1/0]
C 10.21.30.0/24 is directly connected, VLAN2130
C 10.21.40.0/24 is directly connected, VLAN2140
C 192.168.9.0/24 is directly connected, port1
C 192.168.10.0/24 is directly connected, port2

I ran sniffer on FW2 to capture the traffic and this is all no icmp reply

FW2 # diagnose sniffer packet any 'host 192.168.9.181' 4 0 1 interfaces=[any]
Using Original Sniffing Mode
interfaces=[any]
filters=[host 192.168.9.181]
pcap_snapshot: snaplen raised from 0 to 262144
0.696750 port1 in 192.168.9.181 -> 10.21.30.100: icmp: echo request
1.696916 port1 in 192.168.9.181 -> 10.21.30.100: icmp: echo request
2.697103 port1 in 192.168.9.181 -> 10.21.30.100: icmp: echo request
3.697244 port1 in 192.168.9.181 -> 10.21.30.100: icmp: echo request
4.696819 port1 in arp who-has 192.168.9.182 tell 192.168.9.181
4.696838 port1 out arp reply 192.168.9.182 is-at 00:0c:29:a3:9f:f6
25.959741 port1 in 192.168.9.181.10737 -> 192.168.9.182.179: psh 1515141430 ack 1109234650
25.959852 port1 out 192.168.9.182.179 -> 192.168.9.181.10737: ack 1515141449
26.637436 port1 out 192.168.9.182.179 -> 192.168.9.181.10737: psh 1109234650 ack 1515141449
26.638116 port1 in 192.168.9.181.10737 -> 192.168.9.182.179: ack 1109234669
30.977359 port1 out arp who-has 192.168.9.181 tell 192.168.9.182
30.977987 port1 in arp reply 192.168.9.181 is-at 00:0c:29:ef:3e:b4

The policies are the same on both.

Not sure what is going on, any thoughts ?

Thank you

[ede_pfau]

Are port1, port2 on both FGTs allowed to ping?

Apart from the ping experiment:

A FGT is 1- a router AND 2- a firewall. Are the policies sufficient? Which ones have you implemented?

[DPadula]

Do you have administrative access for ping enable on firewall 2 interface with IP address 10.21.30.100?

[srajeswaran]

Can you check if 10.21.30.100 is configured as part of any VIP?

show full-configuration | grep 10.21.30.100 , use "-f" to see the full configuration hierarchy "10.21.30.100 -f"
If there is no references we may run a debug flow to understand the processing.

diagnose debug flow filter daddr 10.21.30.100
diagnose debug flow filter saddr 192.168.9.181
diagnose debug flow filter proto 1
diagnose debug flow show function-name enable
diagnose debug flow show iprope enable
diagnose debug flow trace start 100
diagnose debug enable

[huud]

@ede_pfau and @DPadula all interfaces have ping allowed.

@srajeswaran there is special configuration on the firewall, its all basic configuration.

More Information

FW1 interface

FW1 Policy

FW2 Interface

FW2 Policy

[srajeswaran]

Can you collect the flow debug and share?

[huud]

I ran this on FW2..

FW2 # diagnose debug flow filter daddr 10.21.30.100

FW2 # diagnose debug flow filter saddr 192.168.9.181

FW2 # diagnose debug flow filter proto 1

FW2 # diagnose debug flow show function-name enable
show function name

FW2 # diagnose debug flow show iprope enable
show trace messages about iprope

FW2 # diagnose debug flow trace start 100

FW2 # diagnose debug enable

FW2 # id=65308 trace_id=1 func=print_pkt_detail line=5885 msg="vd-root:0 received a packet(proto=1, 192.168.9.181:0->10.21.30.100:2048) tun_id=0.0.0.0 from port1. type=8, code=0, id=0, seq=0."
id=65308 trace_id=1 func=init_ip_session_common line=6071 msg="allocate a new session-0000012e, tun_id=0.0.0.0"
id=65308 trace_id=1 func=iprope_dnat_check line=5459 msg="in-[port1], out-[]"
id=65308 trace_id=1 func=iprope_dnat_tree_check line=824 msg="len=0"
id=65308 trace_id=1 func=iprope_dnat_check line=5480 msg="result: skb_flags-02000000, vid-0, ret-no-match, act-accept, flag-00000000"
id=65308 trace_id=1 func=__vf_ip_route_input_rcu line=1999 msg="find a route: flag=80000000 gw-0.0.0.0 via root"
id=65308 trace_id=1 func=iprope_access_proxy_check line=458 msg="in-[port1], out-[], skb_flags-02000000, vid-0"
id=65308 trace_id=1 func=__iprope_check line=2391 msg="gnum-100017, check-000000004d58f73b"
id=65308 trace_id=1 func=iprope_policy_group_check line=4886 msg="after check: ret-no-match, act-accept, flag-00000000, flag2-00000000"
id=65308 trace_id=1 func=__iprope_fwd_check line=801 msg="in-[port1], out-[VLAN2130], skb_flags-02000000, vid-0, app_id: 0, url_cat_id: 0"
id=65308 trace_id=1 func=__iprope_tree_check line=524 msg="gnum-100004, use int hash, slot=35, len=2"
id=65308 trace_id=1 func=__iprope_check_one_policy line=2128 msg="checked gnum-100004 policy-3, ret-no-match, act-accept"
id=65308 trace_id=1 func=__iprope_check_one_policy line=2128 msg="checked gnum-100004 policy-0, ret-matched, act-accept"
id=65308 trace_id=1 func=__iprope_user_identity_check line=1891 msg="ret-matched"
id=65308 trace_id=1 func=__iprope_check_one_policy line=2361 msg="policy-0 is matched, act-drop"
id=65308 trace_id=1 func=__iprope_fwd_check line=838 msg="after iprope_captive_check(): is_captive-0, ret-matched, act-drop, idx-0"
id=65308 trace_id=1 func=fw_local_in_handler line=616 msg="iprope_in_check() check failed on policy 0, drop"
id=65308 trace_id=2 func=print_pkt_detail line=5885 msg="vd-root:0 received a packet(proto=1, 192.168.9.181:0->10.21.30.100:2048) tun_id=0.0.0.0 from port1. type=8, code=0, id=0, seq=1."
id=65308 trace_id=2 func=init_ip_session_common line=6071 msg="allocate a new session-0000012f, tun_id=0.0.0.0"
id=65308 trace_id=2 func=iprope_dnat_check line=5459 msg="in-[port1], out-[]"
id=65308 trace_id=2 func=iprope_dnat_tree_check line=824 msg="len=0"
id=65308 trace_id=2 func=iprope_dnat_check line=5480 msg="result: skb_flags-02000000, vid-0, ret-no-match, act-accept, flag-00000000"
id=65308 trace_id=2 func=__vf_ip_route_input_rcu line=1999 msg="find a route: flag=80000000 gw-0.0.0.0 via root"
id=65308 trace_id=2 func=iprope_access_proxy_check line=458 msg="in-[port1], out-[], skb_flags-02000000, vid-0"
id=65308 trace_id=2 func=__iprope_check line=2391 msg="gnum-100017, check-000000004d58f73b"
id=65308 trace_id=2 func=iprope_policy_group_check line=4886 msg="after check: ret-no-match, act-accept, flag-00000000, flag2-00000000"
id=65308 trace_id=2 func=__iprope_fwd_check line=801 msg="in-[port1], out-[VLAN2130], skb_flags-02000000, vid-0, app_id: 0, url_cat_id: 0"
id=65308 trace_id=2 func=__iprope_tree_check line=524 msg="gnum-100004, use int hash, slot=35, len=2"
id=65308 trace_id=2 func=__iprope_check_one_policy line=2128 msg="checked gnum-100004 policy-3, ret-no-match, act-accept"
id=65308 trace_id=2 func=__iprope_check_one_policy line=2128 msg="checked gnum-100004 policy-0, ret-matched, act-accept"
id=65308 trace_id=2 func=__iprope_user_identity_check line=1891 msg="ret-matched"
id=65308 trace_id=2 func=__iprope_check_one_policy line=2361 msg="policy-0 is matched, act-drop"
id=65308 trace_id=2 func=__iprope_fwd_check line=838 msg="after iprope_captive_check(): is_captive-0, ret-matched, act-drop, idx-0"
id=65308 trace_id=2 func=fw_local_in_handler line=616 msg="iprope_in_check() check failed on policy 0, drop"
id=65308 trace_id=3 func=print_pkt_detail line=5885 msg="vd-root:0 received a packet(proto=1, 192.168.9.181:0->10.21.30.100:2048) tun_id=0.0.0.0 from port1. type=8, code=0, id=0, seq=2."
id=65308 trace_id=3 func=init_ip_session_common line=6071 msg="allocate a new session-00000130, tun_id=0.0.0.0"
id=65308 trace_id=3 func=iprope_dnat_check line=5459 msg="in-[port1], out-[]"
id=65308 trace_id=3 func=iprope_dnat_tree_check line=824 msg="len=0"
id=65308 trace_id=3 func=iprope_dnat_check line=5480 msg="result: skb_flags-02000000, vid-0, ret-no-match, act-accept, flag-00000000"
id=65308 trace_id=3 func=__vf_ip_route_input_rcu line=1999 msg="find a route: flag=80000000 gw-0.0.0.0 via root"
id=65308 trace_id=3 func=iprope_access_proxy_check line=458 msg="in-[port1], out-[], skb_flags-02000000, vid-0"
id=65308 trace_id=3 func=__iprope_check line=2391 msg="gnum-100017, check-000000004d58f73b"
id=65308 trace_id=3 func=iprope_policy_group_check line=4886 msg="after check: ret-no-match, act-accept, flag-00000000, flag2-00000000"
id=65308 trace_id=3 func=__iprope_fwd_check line=801 msg="in-[port1], out-[VLAN2130], skb_flags-02000000, vid-0, app_id: 0, url_cat_id: 0"
id=65308 trace_id=3 func=__iprope_tree_check line=524 msg="gnum-100004, use int hash, slot=35, len=2"
id=65308 trace_id=3 func=__iprope_check_one_policy line=2128 msg="checked gnum-100004 policy-3, ret-no-match, act-accept"
id=65308 trace_id=3 func=__iprope_check_one_policy line=2128 msg="checked gnum-100004 policy-0, ret-matched, act-accept"
id=65308 trace_id=3 func=__iprope_user_identity_check line=1891 msg="ret-matched"
id=65308 trace_id=3 func=__iprope_check_one_policy line=2361 msg="policy-0 is matched, act-drop"
id=65308 trace_id=3 func=__iprope_fwd_check line=838 msg="after iprope_captive_check(): is_captive-0, ret-matched, act-drop, idx-0"
id=65308 trace_id=3 func=fw_local_in_handler line=616 msg="iprope_in_check() check failed on policy 0, drop"
id=65308 trace_id=4 func=print_pkt_detail line=5885 msg="vd-root:0 received a packet(proto=1, 192.168.9.181:0->10.21.30.100:2048) tun_id=0.0.0.0 from port1. type=8, code=0, id=0, seq=3."
id=65308 trace_id=4 func=init_ip_session_common line=6071 msg="allocate a new session-00000131, tun_id=0.0.0.0"
id=65308 trace_id=4 func=iprope_dnat_check line=5459 msg="in-[port1], out-[]"
id=65308 trace_id=4 func=iprope_dnat_tree_check line=824 msg="len=0"
id=65308 trace_id=4 func=iprope_dnat_check line=5480 msg="result: skb_flags-02000000, vid-0, ret-no-match, act-accept, flag-00000000"
id=65308 trace_id=4 func=__vf_ip_route_input_rcu line=1999 msg="find a route: flag=80000000 gw-0.0.0.0 via root"
id=65308 trace_id=4 func=iprope_access_proxy_check line=458 msg="in-[port1], out-[], skb_flags-02000000, vid-0"
id=65308 trace_id=4 func=__iprope_check line=2391 msg="gnum-100017, check-000000004d58f73b"
id=65308 trace_id=4 func=iprope_policy_group_check line=4886 msg="after check: ret-no-match, act-accept, flag-00000000, flag2-00000000"
id=65308 trace_id=4 func=__iprope_fwd_check line=801 msg="in-[port1], out-[VLAN2130], skb_flags-02000000, vid-0, app_id: 0, url_cat_id: 0"
id=65308 trace_id=4 func=__iprope_tree_check line=524 msg="gnum-100004, use int hash, slot=35, len=2"
id=65308 trace_id=4 func=__iprope_check_one_policy line=2128 msg="checked gnum-100004 policy-3, ret-no-match, act-accept"
id=65308 trace_id=4 func=__iprope_check_one_policy line=2128 msg="checked gnum-100004 policy-0, ret-matched, act-accept"
id=65308 trace_id=4 func=__iprope_user_identity_check line=1891 msg="ret-matched"
id=65308 trace_id=4 func=__iprope_check_one_policy line=2361 msg="policy-0 is matched, act-drop"
id=65308 trace_id=4 func=__iprope_fwd_check line=838 msg="after iprope_captive_check(): is_captive-0, ret-matched, act-drop, idx-0"
id=65308 trace_id=4 func=fw_local_in_handler line=616 msg="iprope_in_check() check failed on policy 0, drop"
id=65308 trace_id=5 func=print_pkt_detail line=5885 msg="vd-root:0 received a packet(proto=1, 192.168.9.181:0->10.21.30.100:2048) tun_id=0.0.0.0 from port1. type=8, code=0, id=0, seq=4."
id=65308 trace_id=5 func=init_ip_session_common line=6071 msg="allocate a new session-00000132, tun_id=0.0.0.0"
id=65308 trace_id=5 func=iprope_dnat_check line=5459 msg="in-[port1], out-[]"
id=65308 trace_id=5 func=iprope_dnat_tree_check line=824 msg="len=0"
id=65308 trace_id=5 func=iprope_dnat_check line=5480 msg="result: skb_flags-02000000, vid-0, ret-no-match, act-accept, flag-00000000"
id=65308 trace_id=5 func=__vf_ip_route_input_rcu line=1999 msg="find a route: flag=80000000 gw-0.0.0.0 via root"
id=65308 trace_id=5 func=iprope_access_proxy_check line=458 msg="in-[port1], out-[], skb_flags-02000000, vid-0"
id=65308 trace_id=5 func=__iprope_check line=2391 msg="gnum-100017, check-000000004d58f73b"
id=65308 trace_id=5 func=iprope_policy_group_check line=4886 msg="after check: ret-no-match, act-accept, flag-00000000, flag2-00000000"
id=65308 trace_id=5 func=__iprope_fwd_check line=801 msg="in-[port1], out-[VLAN2130], skb_flags-02000000, vid-0, app_id: 0, url_cat_id: 0"
id=65308 trace_id=5 func=__iprope_tree_check line=524 msg="gnum-100004, use int hash, slot=35, len=2"
id=65308 trace_id=5 func=__iprope_check_one_policy line=2128 msg="checked gnum-100004 policy-3, ret-no-match, act-accept"
id=65308 trace_id=5 func=__iprope_check_one_policy line=2128 msg="checked gnum-100004 policy-0, ret-matched, act-accept"
id=65308 trace_id=5 func=__iprope_user_identity_check line=1891 msg="ret-matched"
id=65308 trace_id=5 func=__iprope_check_one_policy line=2361 msg="policy-0 is matched, act-drop"
id=65308 trace_id=5 func=__iprope_fwd_check line=838 msg="after iprope_captive_check(): is_captive-0, ret-matched, act-drop, idx-0"
id=65308 trace_id=5 func=fw_local_in_handler line=616 msg="iprope_in_check() check failed on policy 0, drop"

[srajeswaran]

Its dropped as no policy match found.

id=65308 trace_id=1 func=fw_local_in_handler line=616 msg="iprope_in_check() check failed on policy 0, drop"

Can you share output of below.
show full-configuration | grep 10.21.30.100 -f

[huud]

FW2 # show full-configuration | grep 10.21.30.100 -f
config system interface
    edit "VLAN2130"
        set vdom "root"
        set vrf 0
        set mode static
        set dhcp-relay-interface-select-method auto
        set dhcp-relay-service disable
        set ip 10.21.30.100 255.255.255.0 <---
        set allowaccess ping
        set fail-detect disable
        set pptp-client disable
        set arpforward enable
        set broadcast-forward disable
        set bfd global
        set l2forward disable
        set icmp-send-redirect enable
        set icmp-accept-redirect enable
        set reachable-time 30000
        set vlanforward disable
        set stpforward disable
        set ips-sniffer-mode disable
        set ident-accept disable
        set ipmac disable
        set subst disable
        set substitute-dst-mac 00:00:00:00:00:00
        set status up
        set netbios-forward disable
        set wins-ip 0.0.0.0
        set type vlan
        set netflow-sampler disable
        set sflow-sampler disable
        set src-check enable
        set sample-rate 2000
        set polling-interval 20
        set sample-direction both
        set explicit-web-proxy disable
        set explicit-ftp-proxy disable
        set proxy-captive-portal disable
        set tcp-mss 0
        set inbandwidth 0
        set outbandwidth 0
        set egress-shaping-profile ''
        set ingress-shaping-profile ''
        set spillover-threshold 0
        set ingress-spillover-threshold 0
        set weight 0
        set external disable
        set vlan-protocol 8021q
        set description ''
        set alias ''
        set security-mode none
        set ike-saml-server ''
        set device-identification enable
        set device-user-identification enable
        set estimated-upstream-bandwidth 0
        set estimated-downstream-bandwidth 0
        set measured-upstream-bandwidth 0
        set measured-downstream-bandwidth 0
        set bandwidth-measure-time 0
        set monitor-bandwidth disable
        set vrrp-virtual-mac disable
        set role lan
        set snmp-index 7
        set secondary-IP disable
        set preserve-session-route disable
        set auto-auth-extension-device disable
        set ap-discover enable
        set ip-managed-by-fortiipam inherit-global
        set switch-controller-igmp-snooping-proxy disable
        set switch-controller-igmp-snooping-fast-leave disable
        set switch-controller-feature none
        set switch-controller-offload disable
        set switch-controller-offload-gw disable
        set color 0
        set eap-supplicant disable
        set default-purdue-level 3
        config ipv6
            set ip6-mode static
            set nd-mode basic
            set ip6-address ::/0
            unset ip6-allowaccess
            set icmp6-send-redirect enable
            set ra-send-mtu enable
            set ip6-reachable-time 0
            set ip6-retrans-time 0
            set ip6-hop-limit 0
            set dhcp6-prefix-delegation disable
            set dhcp6-information-request disable
            set vrrp-virtual-mac6 disable
            set vrip6_link_local ::
            set ip6-send-adv disable
            set autoconf disable
            set dhcp6-relay-service disable
        end
        set priority 1
        set dhcp-relay-source-ip 0.0.0.0
        set dhcp-relay-circuit-id ''
        set dhcp-client-identifier ''
        set dhcp-renew-time 0
        set idle-timeout 0
        set disc-retry-timeout 1
        set padt-retry-timeout 1
        set dns-server-override enable
        set dns-server-protocol cleartext
        set wccp disable
        set drop-overlapped-fragment disable
        set drop-fragment disable
        set interface "port1"
        set mtu-override disable
        set vlanid 2130
    next
end

[huud]

Seems like a licensing issue, as both are VM in a test environment I set the same license on both, created a new VM with same settings and a different license, and seems to work without issues..

FW1 # execute ping 10.21.30.100
PING 10.21.30.100 (10.21.30.100): 56 data bytes
64 bytes from 10.21.30.100: icmp_seq=0 ttl=255 time=4.8 ms

Connection lost. Press Enter to start a new session.

FW1 # execute ping 10.21.40.100
PING 10.21.40.100 (10.21.40.100): 56 data bytes
64 bytes from 10.21.40.100: icmp_seq=0 ttl=255 time=0.7 ms

FW3 # execute ping 10.11.30.100
PING 10.11.30.100 (10.11.30.100): 56 data bytes
64 bytes from 10.11.30.100: icmp_seq=0 ttl=255 time=1.0 ms

Connection lost. Press Enter to start a new session.

FW3 # execute ping 10.11.40.100
PING 10.11.40.100 (10.11.40.100): 56 data bytes
64 bytes from 10.11.40.100: icmp_seq=0 ttl=255 time=0.9 ms