Connectivity Issue Between 2 Fortigate ?!

[huud]

Hi,

I have 2 Fortigate setup as below, and the issue as follows.

FW1 cannot ping anything on FW2. network, but FW2 can ping FW1's 10.11.30 and 10.11.40 network.

BGP is setup between the 2 and is working fine, with RouteMap and Prefix List, so traffic from 10.11.30.0 flows through port1 (192.168.9.181) and 10.11.40.0 through port2 (192.168.9.182), smae configuration is on FW2 for 10.21 network.

FW1

Routing table for VRF=0
C 10.11.30.0/24 is directly connected, VLAN30
C 10.11.40.0/24 is directly connected, VLAN40
B 10.21.30.0/24 [20/0] via 192.168.9.182 (recursive is directly connected, port1), 00:18:07, [1/0]
B 10.21.40.0/24 [20/0] via 192.168.10.182 (recursive is directly connected, port2), 00:17:40, [1/0]
C 192.168.9.0/24 is directly connected, port1
C 192.168.10.0/24 is directly connected, port2

FW2

Routing table for VRF=0
B 10.11.30.0/24 [20/0] via 192.168.9.181 (recursive is directly connected, port1), 00:18:47, [1/0]
B 10.11.40.0/24 [20/0] via 192.168.10.181 (recursive is directly connected, port2), 00:18:15, [1/0]
C 10.21.30.0/24 is directly connected, VLAN2130
C 10.21.40.0/24 is directly connected, VLAN2140
C 192.168.9.0/24 is directly connected, port1
C 192.168.10.0/24 is directly connected, port2

I ran sniffer on FW2 to capture the traffic and this is all no icmp reply

FW2 # diagnose sniffer packet any 'host 192.168.9.181' 4 0 1 interfaces=[any]
Using Original Sniffing Mode
interfaces=[any]
filters=[host 192.168.9.181]
pcap_snapshot: snaplen raised from 0 to 262144
0.696750 port1 in 192.168.9.181 -> 10.21.30.100: icmp: echo request
1.696916 port1 in 192.168.9.181 -> 10.21.30.100: icmp: echo request
2.697103 port1 in 192.168.9.181 -> 10.21.30.100: icmp: echo request
3.697244 port1 in 192.168.9.181 -> 10.21.30.100: icmp: echo request
4.696819 port1 in arp who-has 192.168.9.182 tell 192.168.9.181
4.696838 port1 out arp reply 192.168.9.182 is-at 00:0c:29:a3:9f:f6
25.959741 port1 in 192.168.9.181.10737 -> 192.168.9.182.179: psh 1515141430 ack 1109234650
25.959852 port1 out 192.168.9.182.179 -> 192.168.9.181.10737: ack 1515141449
26.637436 port1 out 192.168.9.182.179 -> 192.168.9.181.10737: psh 1109234650 ack 1515141449
26.638116 port1 in 192.168.9.181.10737 -> 192.168.9.182.179: ack 1109234669
30.977359 port1 out arp who-has 192.168.9.181 tell 192.168.9.182
30.977987 port1 in arp reply 192.168.9.181 is-at 00:0c:29:ef:3e:b4

The policies are the same on both.

Not sure what is going on, any thoughts ?

Thank you

[ede_pfau]

I meant, on the VMs, their gateway should always be the FGT port they are connected to - if in VLAN2130, for example, it should be 10.21.30.100.

[huud]

This is exactly hos it is configured..

I ran debug on both firewalls, on FW1 I'm seeing..

id=65308 trace_id=2 func=print_pkt_detail line=5885 msg="vd-root:0 received a packet(proto=1, 10.11.30.200:57185->10.21.40.100:2048) tun_id=0.0.0.0 from VLAN30. type=8, code=0, id=57185, seq=2."
id=65308 trace_id=2 func=resolve_ip_tuple_fast line=5973 msg="Find an existing session, id-00000016, original direction"
id=65308 trace_id=2 func=ipv4_fast_cb line=53 msg="enter fast path"
id=65308 trace_id=3 func=print_pkt_detail line=5885 msg="vd-root:0 received a packet(proto=1, 10.11.30.200:57185->10.21.40.100:2048) tun_id=0.0.0.0 from VLAN30. type=8, code=0, id=57185, seq=3."
id=65308 trace_id=3 func=resolve_ip_tuple_fast line=5973 msg="Find an existing session, id-00000016, original direction"

On FW2 I'm seeing..

id=65308 trace_id=1 func=iprope_dnat_check line=5480 msg="result: skb_flags-02000000, vid-0, ret-no-match, act-accept, flag-00000000"
id=65308 trace_id=1 func=ip_route_input_slow line=1706 msg="reverse path check fail, drop"
id=65308 trace_id=1 func=ip_session_handle_no_dst line=6157 msg="trace"
id=65308 trace_id=2 func=print_pkt_detail line=5885 msg="vd-root:0 received a packet(proto=1, 10.11.30.200:57185->10.21.40.100:2048) tun_id=0.0.0.0 from port2. type=8, code=0, id=57185, seq=15."
id=65308 trace_id=2 func=init_ip_session_common line=6071 msg="allocate a new session-00000022, tun_id=0.0.0.0"
id=65308 trace_id=2 func=iprope_dnat_check line=5459 msg="in-[port2], out-[]"
id=65308 trace_id=2 func=iprope_dnat_tree_check line=824 msg="len=0"
id=65308 trace_id=2 func=iprope_dnat_check line=5480 msg="result: skb_flags-02000000, vid-0, ret-no-match, act-accept, flag-00000000"
id=65308 trace_id=2 func=ip_route_input_slow line=1706 msg="reverse path check fail, drop"

The trace was from for ping from 10.11.30.200 (VM on FW1) to 10.21.40.200 (VM on FW2 on port 2)

[huud]

Tha tis correct, the VM has the FW1 interface as its gateway.

For some reason my other reply is gone for moderation, it has output form the firewalls showing reverse path check fail, drop error. I looked it up and seems like this happens when there is no route back, I thought BGP would have the route back in the routing table.

[huud]

Thanks everyone, and @ede_pfau 

I finally found error to be due to the issue to be Symmetric and Asymmetric routing, and changed the configuration on both routers with the below.

config system settings
    set auxiliary-session enable
end

And finally with NAT disabled, and auxiliary-session enabled, the result is all interfaces are reachable by VMs behind both the firewalls.

[ede_pfau]

Nice find, and I'm glad your setup is working.

Still there seems to be a better way to handle this. If both ports are used interchangeably, you could combine them into one 'zone'. This zone object will then be used in policies instead of an interface.

Usually, zones group interfaces which will be governed by similar policies. (thus, zones only exist in the policy table, not in routing etc.). Grouping interfaces will significantly reduce the number of policies, making the policy table less cluttered and more robust.

I can imagine that in your case it would help with session rerouting, as there will only 1 session created originating from or going to a zone. Thus, the session table doesn't have to be rebuilt as often, as enabled by the "auxiliary-session" parameter.

I am away from my lab but if you feel like it would be a good idea you might try. Please note that in order to pull an interface into a (new) zone, the interface needs to be unused by any other construct (= reference counter is 0).

One way to achieve this with a working config is to get the backup in plain text, and search&replace the interface names with the zone name (e.g., "port1" with "WANzone"). And then restore the config, causing a reboot.

Of course, you could as well leave it as is if the effort would be too high. You are lucky that the parameter exists, it was only introduced in v6.2.3 and is rarely used IMHO.

[huud]

Thanks for elaborating, will need some time to understand, and just so I don't misunderstand can I know if combining interfaces into zones is an alternate to auxiliary-sessions setting ?

[ede_pfau]

If return traffic alternates between the originating interface and a second one, as is the case in your setup, then you need an aux session to mediate it. IMHO "asym routing" has nothing to do with it.

What my idea is that using a zone will take away the ambiguity between interfaces, and thus aux sessions won't be necessary anymore. Besides, it halves the number of policies.

I don't want to talk you into anything. You might as well wait for other responses in this thread, or ask Support via ticket.

[huud]

No worries @ede_pfau 

I actually like to tinker things to learn, will try this it understanding, you definitely have been a big help, appreciate it very much..