SD-WAN rule matching traffic not routed to SD-WAN interface

[Agent_1994]

Hello,

 I've just had a problem at a customer's site. Here's the relevant part of the setup:

[ul]

4 WAN connections, all on a volume-based SD-WAN interface, all with equal weight (25).

Default gateway for that VDOM is the SD-WAN interface.

No policy routing.

There's a Exchange SMTP Virtual IP on wan03.

All outgoing mail must leave on wan03. In order to do this, i created a SD-WAN rule:[ul]

Source: all

Destination: Address, all.

Protocol: TCP

Port Range: 25-25

Interface members: wan03.[/ul][/ul]

 What happened? Everything was fine for outbound traffic, but not for incoming traffic. By doing a diagnose i found that FortiGate tried to route traffic destined to the internal IP of the Exchange server back to the wan03 interface due to policy routing:

id=20085 trace_id=2 func=print_pkt_detail line=5363 msg="vd-firewall received a packet(proto=6, 40.92.4.61:45380->EXTERNAL_IP_WAN03:25) from wan03.  flag S, seq 3694663655, ack 0, win 8192" id=20085 trace_id=2 func=init_ip_session_common line=5519 msg="allocate a new session-010b3d57" id=20085 trace_id=2 func=iprope_dnat_check line=4777 msg="in-[wan03], out-[]" id=20085 trace_id=2 func=iprope_dnat_tree_check line=838 msg="len=1" id=20085 trace_id=2 func=__iprope_check_one_dnat_policy line=4651 msg="checking gnum-100000 policy-1" id=20085 trace_id=2 func=get_new_addr line=1039 msg="find DNAT: IP-INTERNAL_IP_EXCHANGE, port-25" id=20085 trace_id=2 func=__iprope_check_one_dnat_policy line=4733 msg="matched policy-1, act=accept, vip=1, flag=100, sflag=2000000" id=20085 trace_id=2 func=iprope_dnat_check line=4790 msg="result: skb_flags-02000000, vid-1, ret-matched, act-accept, flag-00000100" id=20085 trace_id=2 func=fw_pre_route_handler line=184 msg="VIP-INTERNAL_IP_EXCHANGE:25, outdev-wan03" id=20085 trace_id=2 func=__ip_session_run_tuple line=3178 msg="DNAT EXTERNAL_IP_WAN03:25->INTERNAL_IP_EXCHANGE:25" id=20085 trace_id=2 func=vf_ip_route_input_common line=2574 msg="Match policy routing: to INTERNAL_IP_EXCHANGE via ifindex-43" id=20085 trace_id=2 func=vf_ip_route_input_common line=2583 msg="find a route: flag=00000000 gw-GATEWAY_WAN03 via wan03" id=20085 trace_id=2 func=iprope_fwd_check line=673 msg="in-[wan03], out-[wan03], skb_flags-020000c0, vid-1, app_id: 0, url_cat_id: 0" id=20085 trace_id=2 func=__iprope_tree_check line=545 msg="gnum-100004, use addr/intf hash, len=5" id=20085 trace_id=2 func=__iprope_check_one_policy line=1878 msg="checked gnum-100004 policy-1, ret-no-match, act-accept" id=20085 trace_id=2 func=__iprope_check_one_policy line=1878 msg="checked gnum-100004 policy-2, ret-no-match, act-accept" id=20085 trace_id=2 func=__iprope_check_one_policy line=1878 msg="checked gnum-100004 policy-3, ret-no-match, act-accept" id=20085 trace_id=2 func=__iprope_check_one_policy line=1878 msg="checked gnum-100004 policy-4, ret-no-match, act-accept" id=20085 trace_id=2 func=__iprope_check_one_policy line=1878 msg="checked gnum-100004 policy-0, ret-matched, act-accept" id=20085 trace_id=2 func=__iprope_user_identity_check line=1703 msg="ret-matched" id=20085 trace_id=2 func=__iprope_check_one_policy line=2075 msg="policy-0 is matched, act-drop" id=20085 trace_id=2 func=iprope_fwd_auth_check line=729 msg="after iprope_captive_check(): is_captive-0, ret-matched, act-drop, idx-0" id=20085 trace_id=2 func=iprope_shaping_check line=823 msg="in-[wan03], out-[wan03], skb_flags-020000c0, vid-1" id=20085 trace_id=2 func=__iprope_check line=2104 msg="gnum-100015, check-ffffffffa0020a00" id=20085 trace_id=2 func=__iprope_check_one_policy line=1878 msg="checked gnum-100015 policy-1, ret-no-match, act-accept" id=20085 trace_id=2 func=__iprope_check line=2123 msg="gnum-100015 check result: ret-no-match, act-accept, flag-00000000, flag2-00000000" id=20085 trace_id=2 func=iprope_policy_group_check line=4193 msg="after check: ret-no-match, act-accept, flag-00000000, flag2-00000000" id=20085 trace_id=2 func=fw_forward_handler line=586 msg="Denied by forward policy check (policy 0)"

 It seems that the SD-WAN rule matched traffic meant to the internal interface. I fixed it, or rather worked around it, by specifying the source address for the Exchange server in the SD-WAN rule. 

 I think that SD-WAN rules should only match traffic going to the SD-WAN interface... or is this expected?

[Uwe_Sommerfeld]

Yes, this is what I really don't like about SD-WAN. The rules seem to be just a retarded Version of the policy Routing engine. However, they even kick in if they shouldn't even be considered. So - Workaround - create policy rules with your internal Network Segments and port 25 and Point that rule to the corresponding Interface. it will kick in before sd-wan Routing and send your traffic to the right Interface.

If anybody knows better please comment. Thanks.

[Uwe_Sommerfeld]

By the way in my lab the follwing is not working

a) use a VIP with a defined WAN Interface -> the SD WAN Router should then consider the VIP and automatically also consider the linked Interface. Would be nice but doesn't happen. IP Translation will still be done but any SD WAn Interface may be used.

b) create a Network object for 0.0.0.0/0 and bind it to a specific Interface, use that one in your Policy rules. Was

proposed to me as a Workaround but diesn't work in my lab.

so, IMO one Solution is - use policy Routing - use "stop policy Routing" rules for your internal Networks or create rules with Destination / Interface for all internal destinations (ugly) - use policy Routing for sd-wan override wherever needed - don't use SDWAN rules.

Still, comments are welcome.

[Pieterv]

Is there a possibility that this wil be corrected in the near future? I use SD-WAN and SD-WAN rules but by that, the routes from vlan to vlan are not working, everything is going thru the SD-WAN interface. This is very ugly, with the solution from Uwe, the use of "Stop policy routing" i can finally reach my internal VLAN's again.