Central NAT Mode breaking Traffic Flow - Palo Conversion

[Jasys]

Hi, Was hoping someone may have seen this or knows what is happening, I will do my best to explain:

I am in the process of converting legacy firewalls to a VDOM 400F, all has gone successful  except for one conversion from a palo to an Internal VDOM. I used Forti convertor for all the conversions of objects policies etc and put the VDOM into Central NAT to make it easier, (I did some ASA conversions and it converted better this way)

My last conversion Palo  > FGT, all worked except, 1 very important traffic flow, so had to "backout" and reintroduce the Palo, so I cant even do live troubleshooting untill I attempt it again.

Traffic flow is: Azure Cloud Exchange (51.x.x.x) > External VDOM Firewall destination Public Firewall address (60.60.60.1 as an example) 

DNAT on External firewall changes 60.60.60.1 to 192.168.1.1 which is located off the INTERNAL VDOM, This all works fine, and at the moment the traffic passed to the PALO and goes on its merry way. as it leaves the External VDOM, there is a source NAT also (Central Nat) that changes the public source 51.x.x.x to the exit interface, so the Palo sees the source as 10.10.10.1 as an example,

This flow breaks when I convert to the INTERNAL VDOM firewall from the Palo,  it hits the external VDOM as normal, but when it arrives on the Internal VDOM, its instantly dropped despite there being a rule to allow it (its above the deny, and the RULE is 100% correct) it drops with this in the diag debug:

INTERAL-FW (Internal) # diag debug flow trace start
INTERAL-FW (Internal) # id=65308 trace_id=47 func=print_pkt_detail line=6005 msg="vd-Internal:0 received a packet(proto=6, 10.10.10.1:54 m PORT1. flag [S], seq 2471198223, ack 0, win 65535"
id=65308 trace_id=47 func=init_ip_session_common line=6206 msg="allocate a new session-1f7a1fed"
id=65308 trace_id=47 func=iprope_dnat_check line=5487 msg="in-[PORT1], out-[]"
id=65308 trace_id=47 func=iprope_dnat_tree_check line-824 msg="len=0"
id=65308 trace_id=47 func=iprope_dnat_check line=5512 msg="result: skb_flags-02000000, vid-0, ret-no-match, act-accept, flag-00000000"
id=65308 trace_id=47 func=__vf_ip_route_input_rcu line=1989 msg="find a route: flag-80000000 gw-0.0.0.0 via Internal"
id=65308 trace_id=47 func=iprope_access_proxy_check line=457 msg="in-[PORT1T], out-[], skb_flags-02000000, vid-0"
id=65308 trace_id=47 func=__iprope_check line=2410 msg="gnum-100017, check-ffffffffa002c240"
id=65308 trace_id=47 func=iprope_policy_group_check line-4909 msg="after check: ret-no-match, act-accept, flag-00000000, flag2-00000000" id=65308 trace_id=47 func=iprope_in_check line-495 msg="in-[PORT1], out-[], skb_flags-02000000, vid-0"
id=65308 trace_id=47 func=__iprope_check line=2410 msg="gnum-100011, check-ffffffffa002cc70"
id=65308 trace_id=47 func=iprope_policy_group_check line-4909 msg="after check: ret-no-match, act-drop, flag-00000000, flag2-00000000" id=65308 trace_id=47 func=__iprope_check line=2410 msg="gnum-100001, check-ffffffffa002c240"
id=65308 trace_id=47 func=iprope_policy_group_check line-4909 msg="after check: ret-no-match, act-accept, flag-00000000, flag2-00000000" id=65308 trace_id=47 func=__iprope_check line=2410 msg="gnum-10000e, check-ffffffffa002c240"
id=65308 trace_id=47 func=__iprope_check_one_policy line=2146 msg="checked gnum-10000e policy-4294967295, ret-no-match, act-accept" id=65308 trace_id=47 func=__iprope_check_one_policy line=2146 msg="checked gnum-10000e policy-4294967295, ret-no-match, act-accept" id=65308 trace_id=47 func=__iprope_check_one_policy line=2146 msg="checked gnum-10000e policy-4294967295, ret-no-match, act-accept" id=65308 trace_id=47 func=__iprope_check_one_policy line=2146 msg="checked gnum-10000e policy-4294967295, ret-no-match, act-accept" id=65308 trace_id=47 func=__iprope_check_one_policy line=2146 msg="checked gnum-10000e policy-4294967295, ret-matched, act-accept" id=65308 trace_id=47 func=__iprope_check_one_policy line=2380 msg="policy-4294967295 is matched, act-drop"
id=65308 trace_id=47 func=__iprope_check line=2427 msg="gnum-10000e check result: ret-matched, act-drop, flag-00000000, flag2-00000000" id=65308 trace_id=47 func=iprope_policy_group_check line-4909 msg="after check: ret-matched, act-drop, flag-00000000, flag2-00000000" id=65308 trace_id=47 func=__iprope_check line=2410 msg="gnum-10000f, check-ffffffffa002c240"
id=65308 trace_id=47 func=__iprope_check_one_policy line=2146 msg="checked gnum-10000f policy-4294967295, ret-no-match, act-accept" id=65308 trace_id=47 func=__iprope_check_one_policy line=2146 msg="checked gnum-10000f policy-4294967295, ret-no-match, act-accept" id=65308 trace_id=47 func=__iprope_check_one_policy line=2146 msg="checked gnum-10000f policy-4294967295, ret-no-match, act-accept" id=65308 trace_id=47 func=__iprope_check_one_policy line=2146 msg="checked gnum-10000f policy-4294967295, ret-no-match, act-accept" id=65308 trace_id=47 func=__iprope_check_one_policy line=2146 msg="checked gnum-10000f policy-4294967295, ret-matched, act-accept" id=65308 trace_id=47 func=__iprope_check_one_policy line-2380 msg="policy-4294967295 is matched, act-drop"
id=65308 trace_id=47 func=__iprope_check line=2427 msg="gnum-10000f check result: ret-matched, act-drop, flag-00000800, flag2-00000000" id=65308 trace_id=47 func=iprope_policy_group_check line-4909 msg="after check: ret-matched, act-drop, flag-00000800, flag2-00000000" id=65308 trace_id=47 func=fw_local_in_handler line=620 msg="iprope_in_check() check failed on policy 0, drop"

This line:

id=65308 trace_id=47 func=iprope_dnat_check line=5487 msg="in-[PORT1], out-[]"

doesn't show an exit interface? which I think is causing it, the out interface is an EMAC VLAN , and is definitely correct as other traffic is received. My colleague is suggesting the Internal VDOM sees this traffic as local to itself, so drops it? I dont know why!  On the External VDOM, as I said there is a Source NAT which is working as I see the "vd-Internal:0 received a packet(proto=6, 10.10.10.1:54 )" which is the source NAT, now the destination 192.168.1.1 has a DNAT on the External as I said, and it had "set source-nat-vip enable" which I think may be the issue? but dont really understand whats happening.

I hope this makes sense, and would love to resolve this

Thanks

[wmichael]

When set source-nat-vip enable is configured on a VIP and SNAT occurs the VIP's external IP will be used for the SNAT.  Check this article for the behavior.

https://community.fortinet.com/t5/FortiGate/Technical-Tip-How-to-use-VIP-s-External-IP-Address-for-S...

[Jasys]

Im not quite grasping it ;) as I have SNAT for the reversed traffic, should I disable source-nat-vip ? all other VIP traffic works except this VIP, there is bi-directional traffic here, so when its sourced from the internal VDOM going out, it works when cutover to the Fortigate, its just incoming that doesnt

[wmichael]

id=65308 trace_id=47 func=__vf_ip_route_input_rcu line=1989 msg="find a route: flag-80000000 gw-0.0.0.0 via Internal"

This looks like it's routing to itself, it should show that it found a route via the outgoing interface.  In your case it's routing via "Internal" which is the VDOM.  I believe there must be an error in where the NAT is occurring. 

id=65308 trace_id=47 func=print_pkt_detail line=6005 msg="vd-Internal:0 received a packet(proto=6, 10.10.10.1:54 m PORT1. flag [S], seq 2471198223, ack 0, win 65535"

This doesn't show the destination, it should show something like this example:

id=65308 trace_id=571 func=print_pkt_detail line=5942 msg="vd-root:0 received a packet(proto=6, 192.168.20.2:63611->63.137.229.1:443) tun_id=0.0.0.0 from port6. flag [S], seq 3839416144, ack 0, win 64240"

What does the flow trace look like when it leaves the External VDOM?

[Jasys]

Hi, Many thanks for responding, so I cant so any traces at the moment, as I had to cut back to the older firewall (Palo) which is a huge pain! but yes, it looks like its routing to itself, when  it shouldnt as other traffic works just fine, to add a little more! the exit interface, after its done its DNAT to 192.168.1.1 ,

The route is from External VDOM to Internal VDOM, but not via a VDOM link, its an EMAC VLAN,

so the exit interface is (for example) EMAC-EXT 10.10.10.1  and the Internal VDOM has enter interface as EMAC-INT 10.10.10.2.  and thats the routing on the External VDOM, to get to 192.168.1.1 (Post DNAT address) > Next Hop 10.10.10.2 via Interface EMAC-EXT (10.10.10.1)

This seems to happen as there are logs, but then the Internal VDOM drops it immediately. I can see the REAL EXTERNAL IP being hit, then the DNAT to 192.168.1.1 then as it travels to the Internal VDOM its Source NATs to 10.10.10.1 which is what I want,  so, is the internal VDOM seeing the source 10.10.10.1 and says ,"thats my interface" and drops it? why? 

the rule on the internal VDOM says "source 10.10.10.1 > 192.168.1.1 on port 443 = Allow"  then it should go off on its merry way!  its frustrating, as I need to fix this or at least know how to fix it before I cut over again, 

P.S Internal is the name of the Internal VDOM, thats why it says that I guess.

the traffic log says "TRAN DISPLAY snat+dnat"  when logging the rule on the External, so its doing what I expect

[wmichael]

I see I think, I understand the traffic flow a bit better now.

Basically:

51.x.x.x --> 60.60.60.1 --> (DNAT 192.168.1.1)[External VDOM](EMAC-EXT SNAT 10.10.10.1)-->(EMAC-INT 10.10.10.2)[Internal VDOM]

In the Internal VDOM is there any VIP, IP pool, or loopback interface that might cover the IP 10.10.10.1?  I'm trying to think of reasons why it might route to itself.  On a FortiGate VIPs and IP pools are considered local to the device even if they are not used.

[Jasys]

Well! another Fortinet chap did point out that IPPOOLs would cause this behaviour, on my Internal firewall, there is a POOL, that covers 10.10.10.1 (this is needed for the reverse traffic being initiated, and it has ARP-REPLY enabled, I think this is the smoking gun? when traffic arrives at the internal VDOM, it doesnt forward it, as the ippool is telling the gate, I have this address dont forward it? 

[wmichael]

If there is an IP pool covering that IP address, then the FortiGate will consider that IP address local to itself, even if the IP pool is not referenced any where.  The behaviour is the same with VIPs.  I would recommend deleting it because it's very likely that is why the Internal VDOM routed the traffic to itself.

This behaviour is discussed in this article: https://community.fortinet.com/t5/FortiGate/Technical-T...

If 10.10.10.1 is the interface IP address of the EMAC-EXT intercase in the External VDOM why would it be needed for an IP pool in the Internal VDOM?

[Jasys]

The Destination 192.168.1.1 is in an IPPOOL, as traffic can be sourced outbound..so im guessing when its inbound, fortigate sees the pool with arp reply and drops it as local traffic..this would explain what I am seeing.