Explorer II

Solved

Connectivity Issue Between 2 Fortigate ?!

Forum|Forum|2 years ago
June 8, 2024
10 replies
10871 views

Hi,

I have 2 Fortigate setup as below, and the issue as follows.

FW1 cannot ping anything on FW2. network, but FW2 can ping FW1's 10.11.30 and 10.11.40 network.

BGP is setup between the 2 and is working fine, with RouteMap and Prefix List, so traffic from 10.11.30.0 flows through port1 (192.168.9.181) and 10.11.40.0 through port2 (192.168.9.182), smae configuration is on FW2 for 10.21 network.

FW1

Routing table for VRF=0 C 10.11.30.0/24 is directly connected, VLAN30 C 10.11.40.0/24 is directly connected, VLAN40 B 10.21.30.0/24 [20/0] via 192.168.9.182 (recursive is directly connected, port1), 00:18:07, [1/0] B 10.21.40.0/24 [20/0] via 192.168.10.182 (recursive is directly connected, port2), 00:17:40, [1/0] C 192.168.9.0/24 is directly connected, port1 C 192.168.10.0/24 is directly connected, port2

FW2

Routing table for VRF=0 B 10.11.30.0/24 [20/0] via 192.168.9.181 (recursive is directly connected, port1), 00:18:47, [1/0] B 10.11.40.0/24 [20/0] via 192.168.10.181 (recursive is directly connected, port2), 00:18:15, [1/0] C 10.21.30.0/24 is directly connected, VLAN2130 C 10.21.40.0/24 is directly connected, VLAN2140 C 192.168.9.0/24 is directly connected, port1 C 192.168.10.0/24 is directly connected, port2

I ran sniffer on FW2 to capture the traffic and this is all no icmp reply

FW2 # diagnose sniffer packet any 'host 192.168.9.181' 4 0 1 interfaces=[any] Using Original Sniffing Mode interfaces=[any] filters=[host 192.168.9.181] pcap_snapshot: snaplen raised from 0 to 262144 0.696750 port1 in 192.168.9.181 -> 10.21.30.100: icmp: echo request 1.696916 port1 in 192.168.9.181 -> 10.21.30.100: icmp: echo request 2.697103 port1 in 192.168.9.181 -> 10.21.30.100: icmp: echo request 3.697244 port1 in 192.168.9.181 -> 10.21.30.100: icmp: echo request 4.696819 port1 in arp who-has 192.168.9.182 tell 192.168.9.181 4.696838 port1 out arp reply 192.168.9.182 is-at 00:0c:29:a3:9f:f6 25.959741 port1 in 192.168.9.181.10737 -> 192.168.9.182.179: psh 1515141430 ack 1109234650 25.959852 port1 out 192.168.9.182.179 -> 192.168.9.181.10737: ack 1515141449 26.637436 port1 out 192.168.9.182.179 -> 192.168.9.181.10737: psh 1109234650 ack 1515141449 26.638116 port1 in 192.168.9.181.10737 -> 192.168.9.182.179: ack 1109234669 30.977359 port1 out arp who-has 192.168.9.181 tell 192.168.9.182 30.977987 port1 in arp reply 192.168.9.181 is-at 00:0c:29:ef:3e:b4

The policies are the same on both.

Not sure what is going on, any thoughts ?

Thank you

FortiGate

Best answer by ede_pfau

For example, you're missing VL30 > p2, if VL30 is sending traffic to VL2140. You've got only half of the required policies.

ede_pfau

SuperUser

Are port1, port2 on both FGTs allowed to ping?

Apart from the ping experiment:

A FGT is 1- a router AND 2- a firewall. Are the policies sufficient? Which ones have you implemented?

DPadula

Staff & Editor

Do you have administrative access for ping enable on firewall 2 interface with IP address 10.21.30.100?

srajeswaran

Staff

Can you check if 10.21.30.100 is configured as part of any VIP?

show full-configuration | grep 10.21.30.100 , use "-f" to see the full configuration hierarchy "10.21.30.100 -f"
If there is no references we may run a debug flow to understand the processing.

diagnose debug flow filter daddr 10.21.30.100
diagnose debug flow filter saddr 192.168.9.181
diagnose debug flow filter proto 1
diagnose debug flow show function-name enable
diagnose debug flow show iprope enable
diagnose debug flow trace start 100
diagnose debug enable

huudAuthor

Explorer II

@ede_pfau and @DPadula all interfaces have ping allowed.

@srajeswaran there is special configuration on the firewall, its all basic configuration.

More Information

FW1 interface

FW1 Policy

FW2 Interface

FW2 Policy

srajeswaran

Staff

Can you collect the flow debug and share?

huudAuthor

Explorer II

I ran this on FW2..

FW2 # diagnose debug flow filter daddr 10.21.30.100  FW2 # diagnose debug flow filter saddr 192.168.9.181  FW2 # diagnose debug flow filter proto 1  FW2 # diagnose debug flow show function-name enable show function name  FW2 # diagnose debug flow show iprope enable show trace messages about iprope  FW2 # diagnose debug flow trace start 100  FW2 # diagnose debug enable  FW2 # id=65308 trace_id=1 func=print_pkt_detail line=5885 msg="vd-root:0 received a packet(proto=1, 192.168.9.181:0->10.21.30.100:2048) tun_id=0.0.0.0 from port1. type=8, code=0, id=0, seq=0." id=65308 trace_id=1 func=init_ip_session_common line=6071 msg="allocate a new session-0000012e, tun_id=0.0.0.0" id=65308 trace_id=1 func=iprope_dnat_check line=5459 msg="in-[port1], out-[]" id=65308 trace_id=1 func=iprope_dnat_tree_check line=824 msg="len=0" id=65308 trace_id=1 func=iprope_dnat_check line=5480 msg="result: skb_flags-02000000, vid-0, ret-no-match, act-accept, flag-00000000" id=65308 trace_id=1 func=__vf_ip_route_input_rcu line=1999 msg="find a route: flag=80000000 gw-0.0.0.0 via root" id=65308 trace_id=1 func=iprope_access_proxy_check line=458 msg="in-[port1], out-[], skb_flags-02000000, vid-0" id=65308 trace_id=1 func=__iprope_check line=2391 msg="gnum-100017, check-000000004d58f73b" id=65308 trace_id=1 func=iprope_policy_group_check line=4886 msg="after check: ret-no-match, act-accept, flag-00000000, flag2-00000000" id=65308 trace_id=1 func=__iprope_fwd_check line=801 msg="in-[port1], out-[VLAN2130], skb_flags-02000000, vid-0, app_id: 0, url_cat_id: 0" id=65308 trace_id=1 func=__iprope_tree_check line=524 msg="gnum-100004, use int hash, slot=35, len=2" id=65308 trace_id=1 func=__iprope_check_one_policy line=2128 msg="checked gnum-100004 policy-3, ret-no-match, act-accept" id=65308 trace_id=1 func=__iprope_check_one_policy line=2128 msg="checked gnum-100004 policy-0, ret-matched, act-accept" id=65308 trace_id=1 func=__iprope_user_identity_check line=1891 msg="ret-matched" id=65308 trace_id=1 func=__iprope_check_one_policy line=2361 msg="policy-0 is matched, act-drop" id=65308 trace_id=1 func=__iprope_fwd_check line=838 msg="after iprope_captive_check(): is_captive-0, ret-matched, act-drop, idx-0" id=65308 trace_id=1 func=fw_local_in_handler line=616 msg="iprope_in_check() check failed on policy 0, drop" id=65308 trace_id=2 func=print_pkt_detail line=5885 msg="vd-root:0 received a packet(proto=1, 192.168.9.181:0->10.21.30.100:2048) tun_id=0.0.0.0 from port1. type=8, code=0, id=0, seq=1." id=65308 trace_id=2 func=init_ip_session_common line=6071 msg="allocate a new session-0000012f, tun_id=0.0.0.0" id=65308 trace_id=2 func=iprope_dnat_check line=5459 msg="in-[port1], out-[]" id=65308 trace_id=2 func=iprope_dnat_tree_check line=824 msg="len=0" id=65308 trace_id=2 func=iprope_dnat_check line=5480 msg="result: skb_flags-02000000, vid-0, ret-no-match, act-accept, flag-00000000" id=65308 trace_id=2 func=__vf_ip_route_input_rcu line=1999 msg="find a route: flag=80000000 gw-0.0.0.0 via root" id=65308 trace_id=2 func=iprope_access_proxy_check line=458 msg="in-[port1], out-[], skb_flags-02000000, vid-0" id=65308 trace_id=2 func=__iprope_check line=2391 msg="gnum-100017, check-000000004d58f73b" id=65308 trace_id=2 func=iprope_policy_group_check line=4886 msg="after check: ret-no-match, act-accept, flag-00000000, flag2-00000000" id=65308 trace_id=2 func=__iprope_fwd_check line=801 msg="in-[port1], out-[VLAN2130], skb_flags-02000000, vid-0, app_id: 0, url_cat_id: 0" id=65308 trace_id=2 func=__iprope_tree_check line=524 msg="gnum-100004, use int hash, slot=35, len=2" id=65308 trace_id=2 func=__iprope_check_one_policy line=2128 msg="checked gnum-100004 policy-3, ret-no-match, act-accept" id=65308 trace_id=2 func=__iprope_check_one_policy line=2128 msg="checked gnum-100004 policy-0, ret-matched, act-accept" id=65308 trace_id=2 func=__iprope_user_identity_check line=1891 msg="ret-matched" id=65308 trace_id=2 func=__iprope_check_one_policy line=2361 msg="policy-0 is matched, act-drop" id=65308 trace_id=2 func=__iprope_fwd_check line=838 msg="after iprope_captive_check(): is_captive-0, ret-matched, act-drop, idx-0" id=65308 trace_id=2 func=fw_local_in_handler line=616 msg="iprope_in_check() check failed on policy 0, drop" id=65308 trace_id=3 func=print_pkt_detail line=5885 msg="vd-root:0 received a packet(proto=1, 192.168.9.181:0->10.21.30.100:2048) tun_id=0.0.0.0 from port1. type=8, code=0, id=0, seq=2." id=65308 trace_id=3 func=init_ip_session_common line=6071 msg="allocate a new session-00000130, tun_id=0.0.0.0" id=65308 trace_id=3 func=iprope_dnat_check line=5459 msg="in-[port1], out-[]" id=65308 trace_id=3 func=iprope_dnat_tree_check line=824 msg="len=0" id=65308 trace_id=3 func=iprope_dnat_check line=5480 msg="result: skb_flags-02000000, vid-0, ret-no-match, act-accept, flag-00000000" id=65308 trace_id=3 func=__vf_ip_route_input_rcu line=1999 msg="find a route: flag=80000000 gw-0.0.0.0 via root" id=65308 trace_id=3 func=iprope_access_proxy_check line=458 msg="in-[port1], out-[], skb_flags-02000000, vid-0" id=65308 trace_id=3 func=__iprope_check line=2391 msg="gnum-100017, check-000000004d58f73b" id=65308 trace_id=3 func=iprope_policy_group_check line=4886 msg="after check: ret-no-match, act-accept, flag-00000000, flag2-00000000" id=65308 trace_id=3 func=__iprope_fwd_check line=801 msg="in-[port1], out-[VLAN2130], skb_flags-02000000, vid-0, app_id: 0, url_cat_id: 0" id=65308 trace_id=3 func=__iprope_tree_check line=524 msg="gnum-100004, use int hash, slot=35, len=2" id=65308 trace_id=3 func=__iprope_check_one_policy line=2128 msg="checked gnum-100004 policy-3, ret-no-match, act-accept" id=65308 trace_id=3 func=__iprope_check_one_policy line=2128 msg="checked gnum-100004 policy-0, ret-matched, act-accept" id=65308 trace_id=3 func=__iprope_user_identity_check line=1891 msg="ret-matched" id=65308 trace_id=3 func=__iprope_check_one_policy line=2361 msg="policy-0 is matched, act-drop" id=65308 trace_id=3 func=__iprope_fwd_check line=838 msg="after iprope_captive_check(): is_captive-0, ret-matched, act-drop, idx-0" id=65308 trace_id=3 func=fw_local_in_handler line=616 msg="iprope_in_check() check failed on policy 0, drop" id=65308 trace_id=4 func=print_pkt_detail line=5885 msg="vd-root:0 received a packet(proto=1, 192.168.9.181:0->10.21.30.100:2048) tun_id=0.0.0.0 from port1. type=8, code=0, id=0, seq=3." id=65308 trace_id=4 func=init_ip_session_common line=6071 msg="allocate a new session-00000131, tun_id=0.0.0.0" id=65308 trace_id=4 func=iprope_dnat_check line=5459 msg="in-[port1], out-[]" id=65308 trace_id=4 func=iprope_dnat_tree_check line=824 msg="len=0" id=65308 trace_id=4 func=iprope_dnat_check line=5480 msg="result: skb_flags-02000000, vid-0, ret-no-match, act-accept, flag-00000000" id=65308 trace_id=4 func=__vf_ip_route_input_rcu line=1999 msg="find a route: flag=80000000 gw-0.0.0.0 via root" id=65308 trace_id=4 func=iprope_access_proxy_check line=458 msg="in-[port1], out-[], skb_flags-02000000, vid-0" id=65308 trace_id=4 func=__iprope_check line=2391 msg="gnum-100017, check-000000004d58f73b" id=65308 trace_id=4 func=iprope_policy_group_check line=4886 msg="after check: ret-no-match, act-accept, flag-00000000, flag2-00000000" id=65308 trace_id=4 func=__iprope_fwd_check line=801 msg="in-[port1], out-[VLAN2130], skb_flags-02000000, vid-0, app_id: 0, url_cat_id: 0" id=65308 trace_id=4 func=__iprope_tree_check line=524 msg="gnum-100004, use int hash, slot=35, len=2" id=65308 trace_id=4 func=__iprope_check_one_policy line=2128 msg="checked gnum-100004 policy-3, ret-no-match, act-accept" id=65308 trace_id=4 func=__iprope_check_one_policy line=2128 msg="checked gnum-100004 policy-0, ret-matched, act-accept" id=65308 trace_id=4 func=__iprope_user_identity_check line=1891 msg="ret-matched" id=65308 trace_id=4 func=__iprope_check_one_policy line=2361 msg="policy-0 is matched, act-drop" id=65308 trace_id=4 func=__iprope_fwd_check line=838 msg="after iprope_captive_check(): is_captive-0, ret-matched, act-drop, idx-0" id=65308 trace_id=4 func=fw_local_in_handler line=616 msg="iprope_in_check() check failed on policy 0, drop" id=65308 trace_id=5 func=print_pkt_detail line=5885 msg="vd-root:0 received a packet(proto=1, 192.168.9.181:0->10.21.30.100:2048) tun_id=0.0.0.0 from port1. type=8, code=0, id=0, seq=4." id=65308 trace_id=5 func=init_ip_session_common line=6071 msg="allocate a new session-00000132, tun_id=0.0.0.0" id=65308 trace_id=5 func=iprope_dnat_check line=5459 msg="in-[port1], out-[]" id=65308 trace_id=5 func=iprope_dnat_tree_check line=824 msg="len=0" id=65308 trace_id=5 func=iprope_dnat_check line=5480 msg="result: skb_flags-02000000, vid-0, ret-no-match, act-accept, flag-00000000" id=65308 trace_id=5 func=__vf_ip_route_input_rcu line=1999 msg="find a route: flag=80000000 gw-0.0.0.0 via root" id=65308 trace_id=5 func=iprope_access_proxy_check line=458 msg="in-[port1], out-[], skb_flags-02000000, vid-0" id=65308 trace_id=5 func=__iprope_check line=2391 msg="gnum-100017, check-000000004d58f73b" id=65308 trace_id=5 func=iprope_policy_group_check line=4886 msg="after check: ret-no-match, act-accept, flag-00000000, flag2-00000000" id=65308 trace_id=5 func=__iprope_fwd_check line=801 msg="in-[port1], out-[VLAN2130], skb_flags-02000000, vid-0, app_id: 0, url_cat_id: 0" id=65308 trace_id=5 func=__iprope_tree_check line=524 msg="gnum-100004, use int hash, slot=35, len=2" id=65308 trace_id=5 func=__iprope_check_one_policy line=2128 msg="checked gnum-100004 policy-3, ret-no-match, act-accept" id=65308 trace_id=5 func=__iprope_check_one_policy line=2128 msg="checked gnum-100004 policy-0, ret-matched, act-accept" id=65308 trace_id=5 func=__iprope_user_identity_check line=1891 msg="ret-matched" id=65308 trace_id=5 func=__iprope_check_one_policy line=2361 msg="policy-0 is matched, act-drop" id=65308 trace_id=5 func=__iprope_fwd_check line=838 msg="after iprope_captive_check(): is_captive-0, ret-matched, act-drop, idx-0" id=65308 trace_id=5 func=fw_local_in_handler line=616 msg="iprope_in_check() check failed on policy 0, drop"

huudAuthor

Explorer II

Seems like a licensing issue, as both are VM in a test environment I set the same license on both, created a new VM with same settings and a different license, and seems to work without issues..

FW1 # execute ping 10.21.30.100 PING 10.21.30.100 (10.21.30.100): 56 data bytes 64 bytes from 10.21.30.100: icmp_seq=0 ttl=255 time=4.8 ms  Connection lost. Press Enter to start a new session.  FW1 # execute ping 10.21.40.100 PING 10.21.40.100 (10.21.40.100): 56 data bytes 64 bytes from 10.21.40.100: icmp_seq=0 ttl=255 time=0.7 ms  FW3 # execute ping 10.11.30.100 PING 10.11.30.100 (10.11.30.100): 56 data bytes 64 bytes from 10.11.30.100: icmp_seq=0 ttl=255 time=1.0 ms  Connection lost. Press Enter to start a new session.  FW3 # execute ping 10.11.40.100 PING 10.11.40.100 (10.11.40.100): 56 data bytes 64 bytes from 10.11.40.100: icmp_seq=0 ttl=255 time=0.9 ms

huudAuthor

Explorer II

Appreciate everyones input..

On the same topic, unsure if I should open a different thread.

How to allow traffic for the VM behind FW1 to reach VM behind FW2, and vice versa ?

In the above only firewalls are able to reach each other.

srajeswaran

Staff

If you create firewall policies from VLAN interfaces to port1/port2 on both firewalls and enable source NAT, ideally this communication should work without any additional changes.

huudAuthor

Explorer II

I did this, but VM in FW1 10.11.30/0/24 can reach FW2 10.21.30.0/24 but not 10.21.40.0, this is the same vice versa.

huudAuthor

Explorer II

Thanks,

Is this what is being referred to as Source NAT, this in the Policy itself ?

And do I need to enable Preserve Source Port ?

ede_pfau

SuperUser

I recommend not to go this way. SNAT is poor man's routing.

That is, your routing is insufficient. NATting the traffic will conceal this and make it work but it still is incorrect.

On your VMs, is the default route (0.0.0.0/0) pointing to the FGT? If you don't want to use the default route for this, create a specific route to the remote network, with the gateway address as the address of the directly connected FGT interface. The FGT will then route the traffic to remote.

ede_pfau

SuperUser

and to your question about "preserve source port": you supply 1 NAT address, the interface's address. What will happen if you fix the source port as well? What if you open 2 or more connections/sessions?

huudAuthor

Explorer II

Apologies, not a firewall guy..

The Firewall VLAN interfaces 10.11.30.100 on FW1 is the gateway for the VM, and is what is set in the VM, same on the FW2 and VMs connected to it as well.

ede_pfau

SuperUser

and you've got policies between the gateway ports and the VLANs?

for instance, on FW2, between port1 and VLAN2130?

I apologize, the routing seems to be correct from the routing table you show in your request.

huudAuthor

Explorer II

I have added for port and VLAN 2 policies each.

Port1 to VLAN2130

VLAN2130 to Port1 ---> NAT only on this policy

Port2 to VLAN2140

VLAN2140 to Port2 ---> NAT only on this policy

The same on FW1 as well.

huudAuthor

Explorer II

This is how I understood it and implemented on both sides VLAN.

ede_pfauAnswer

SuperUser

For example, you're missing VL30 > p2, if VL30 is sending traffic to VL2140. You've got only half of the required policies.

huudAuthor

Explorer II

Thanks this solves the issue, ping now reaching from both side to both interfaces..

I'm understanding I need to enable NAT on every VLAn interface is that correct ?

huudAuthor

Explorer II

Thanks everyone, and @ede_pfau

I finally found error to be due to the issue to be Symmetric and Asymmetric routing, and changed the configuration on both routers with the below.

config system settings     set auxiliary-session enable end

And finally with NAT disabled, and auxiliary-session enabled, the result is all interfaces are reachable by VMs behind both the firewalls.

ede_pfau

SuperUser

Nice find, and I'm glad your setup is working.

Still there seems to be a better way to handle this. If both ports are used interchangeably, you could combine them into one 'zone'. This zone object will then be used in policies instead of an interface.

Usually, zones group interfaces which will be governed by similar policies. (thus, zones only exist in the policy table, not in routing etc.). Grouping interfaces will significantly reduce the number of policies, making the policy table less cluttered and more robust.

I can imagine that in your case it would help with session rerouting, as there will only 1 session created originating from or going to a zone. Thus, the session table doesn't have to be rebuilt as often, as enabled by the "auxiliary-session" parameter.

I am away from my lab but if you feel like it would be a good idea you might try. Please note that in order to pull an interface into a (new) zone, the interface needs to be unused by any other construct (= reference counter is 0).

One way to achieve this with a working config is to get the backup in plain text, and search&replace the interface names with the zone name (e.g., "port1" with "WANzone"). And then restore the config, causing a reboot.

Of course, you could as well leave it as is if the effort would be too high. You are lucky that the parameter exists, it was only introduced in v6.2.3 and is rarely used IMHO.

huudAuthor

Explorer II

Thanks for elaborating, will need some time to understand, and just so I don't misunderstand can I know if combining interfaces into zones is an alternate to auxiliary-sessions setting ?

Sign up

Login with SSO

Login to the community

Login with SSO