Policy route does not work in VRF setup

[Minotaur]

Hello!

In my network I have a separate VRF 20, and I need to push traffic from certain users in it. VRF 20 has connectivity to internet via interface vlan96-14:

gw (root) # show system interface vlan96-14
config system interface
    edit "vlan96-14"
        set vdom "root"
        set vrf 20
        set ip XX.XX.136.238 255.255.255.248
        set allowaccess ping
        set type emac-vlan
        set src-check disable
        set role wan
        set snmp-index 51
        set interface "vlan96"
    next
end

VRF 20 is connected to main VRF 0 via NPU vlink:

gw (root) # show system interface vrf-main-to-20
config system interface
    edit "vrf-main-to-20"
        set vdom "root"
        set ip 172.16.255.1 255.255.255.252
        set allowaccess ping
        set snmp-index 53
        set interface "npu0_vlink0"
        set vlanid 4020
    next
end
gw (root) # show system interface vrf-20-to-main
config system interface
    edit "vrf-20-to-main"
        set vdom "root"
        set vrf 20
        set ip 172.16.255.2 255.255.255.252
        set allowaccess ping
        set snmp-index 54
        set interface "npu0_vlink1"
        set vlanid 4020
    next
end

VRF 20 has route to internal network to main VRF:

Routing table for VRF=20
S*      0.0.0.0/0 [1/0] via XX.XX.136.233, vlan96-14, [1/0]
S       10.0.0.0/8 [10/0] via 172.16.255.1, vrf-20-to-main, [1/0]
C       172.16.255.0/30 is directly connected, vrf-20-to-main
C       XXX.XXX.136.232/29 is directly connected, vlan96-14

External IP XX.XX.136.238 is pingable from Internet: 

# ping XX.XX.136.238
PING XX.XX.136.238 (XX.XX.136.238): 56 data bytes
64 bytes from XX.XX.136.238: icmp_seq=0 ttl=251 time=14.085 ms
^C
--- XX.XX.136.238 ping statistics ---
1 packets transmitted, 1 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 14.085/14.085/14.085/0.000 ms

There are two policies to allow traffic from main VRF to VRF 20 and further to Internet:

    edit 37
        set name "Test1"
        set uuid 69f9addc-5611-51ef-bded-1f6ee5405f0b
        set srcintf "DMZ"
        set dstintf "vrf-main-to-20"
        set action accept
        set srcaddr "Servers"
        set dstaddr "all"
        set schedule "always"
        set service "ALL"
        set logtraffic all
    next
    edit 38
        set name "Test2"
        set uuid 3a4b7bcc-5613-51ef-b13c-2a66bd81b27d
        set srcintf "vrf-20-to-main"
        set dstintf "Special"
        set action accept
        set srcaddr "Servers"
        set dstaddr "all"
        set schedule "always"
        set service "ALL"
        set logtraffic all
        set nat enable
    next

Now I need to route traffic from some users via VRF 20. So I've added policy route:

gw (root) # show router policy
config router policy
    edit 1
        set srcaddr "Server:NS"
        set dst "9.9.9.9/255.255.255.255"
        set gateway 172.16.255.2
        set output-device "vrf-main-to-20"
    next
end

... and this does not work. Host Server:NS (10.1.1.2) is unable to ping 9.9.9.9:

# ping -c 1 9.9.9.9
PING 9.9.9.9 (9.9.9.9): 56 data bytes

--- 9.9.9.9 ping statistics ---
1 packets transmitted, 0 packets received, 100.0% packet loss

I've enabled debug and trace ICMP packet from 10.1.1.2 to 9.9.9.9:

# diag debug enable
# diag debug flow filter daddr 9.9.9.9
# diag debug flow show function-name enable
# diag debug flow show iprope enable
# diag debug flow trace start 2
[...]
id=65308 trace_id=4 func=print_pkt_detail line=5879 msg="vd-root:0 received a packet(proto=1, 10.1.1.2:1379->9.9.9.9:2048) tun_id=0.0.0.0 from vlan11. type=8, code=0, id=1379, seq=1
."
id=65308 trace_id=4 func=init_ip_session_common line=6063 msg="allocate a new session-00001bf2"
id=65308 trace_id=4 func=iprope_dnat_check line=5474 msg="in-[vlan11], out-[]"
id=65308 trace_id=4 func=iprope_dnat_tree_check line=834 msg="len=0"
id=65308 trace_id=4 func=iprope_dnat_check line=5499 msg="result: skb_flags-02000000, vid-0, ret-no-match, act-accept, flag-00000000"
id=65308 trace_id=4 func=rpdb_srv_match_input line=1158 msg="Match policy routing id=1: to 172.16.255.2 via ifindex-57"
id=65308 trace_id=4 func=vf_ip_route_input_common line=2612 msg="find a route: flag=84000000 gw-172.16.255.2 via root"
id=65308 trace_id=4 func=iprope_access_proxy_check line=458 msg="in-[vlan11], out-[], skb_flags-02000000, vid-0"
id=65308 trace_id=4 func=__iprope_check line=2395 msg="gnum-100017, check-ffffffbffc02c5b4"
id=65308 trace_id=4 func=iprope_policy_group_check line=4892 msg="after check: ret-no-match, act-accept, flag-00000000, flag2-00000000"
id=65308 trace_id=4 func=iprope_in_check line=496 msg="in-[vlan11], out-[], skb_flags-02000000, vid-0"
id=65308 trace_id=4 func=__iprope_check line=2395 msg="gnum-100011, check-ffffffbffc02d640"
id=65308 trace_id=4 func=iprope_policy_group_check line=4892 msg="after check: ret-no-match, act-drop, flag-00000000, flag2-00000000"
id=65308 trace_id=4 func=__iprope_check line=2395 msg="gnum-100001, check-ffffffbffc02c5b4"
id=65308 trace_id=4 func=iprope_policy_group_check line=4892 msg="after check: ret-no-match, act-accept, flag-00000000, flag2-00000000"
id=65308 trace_id=4 func=__iprope_check line=2395 msg="gnum-10000e, check-ffffffbffc02c5b4"
id=65308 trace_id=4 func=__iprope_check_one_policy line=2131 msg="checked gnum-10000e policy-4294967295, ret-no-match, act-accept"
id=65308 trace_id=4 func=__iprope_check_one_policy line=2131 msg="checked gnum-10000e policy-4294967295, ret-no-match, act-accept"
id=65308 trace_id=4 func=__iprope_check_one_policy line=2131 msg="checked gnum-10000e policy-4294967295, ret-no-match, act-accept"
id=65308 trace_id=4 func=__iprope_check_one_policy line=2131 msg="checked gnum-10000e policy-4294967295, ret-no-match, act-accept"
id=65308 trace_id=4 func=__iprope_check_one_policy line=2131 msg="checked gnum-10000e policy-4294967295, ret-no-match, act-accept"
id=65308 trace_id=4 func=__iprope_check_one_policy line=2131 msg="checked gnum-10000e policy-4294967295, ret-no-match, act-accept"
id=65308 trace_id=4 func=__iprope_check_one_policy line=2131 msg="checked gnum-10000e policy-4294967295, ret-no-match, act-accept"
id=65308 trace_id=4 func=__iprope_check_one_policy line=2131 msg="checked gnum-10000e policy-4294967295, ret-no-match, act-accept"
id=65308 trace_id=4 func=__iprope_check_one_policy line=2131 msg="checked gnum-10000e policy-4294967295, ret-no-match, act-accept"
id=65308 trace_id=4 func=__iprope_check_one_policy line=2131 msg="checked gnum-10000e policy-4294967295, ret-no-match, act-accept"
id=65308 trace_id=4 func=__iprope_check_one_policy line=2131 msg="checked gnum-10000e policy-4294967295, ret-no-match, act-accept"
id=65308 trace_id=4 func=__iprope_check_one_policy line=2131 msg="checked gnum-10000e policy-4294967295, ret-no-match, act-accept"
id=65308 trace_id=4 func=__iprope_check_one_policy line=2131 msg="checked gnum-10000e policy-4294967295, ret-no-match, act-accept"
id=65308 trace_id=4 func=__iprope_check_one_policy line=2131 msg="checked gnum-10000e policy-4294967295, ret-no-match, act-accept"
id=65308 trace_id=4 func=__iprope_check_one_policy line=2131 msg="checked gnum-10000e policy-4294967295, ret-no-match, act-accept"
id=65308 trace_id=4 func=__iprope_check_one_policy line=2131 msg="checked gnum-10000e policy-4294967295, ret-no-match, act-accept"
id=65308 trace_id=4 func=__iprope_check_one_policy line=2131 msg="checked gnum-10000e policy-4294967295, ret-no-match, act-accept"
id=65308 trace_id=4 func=__iprope_check_one_policy line=2131 msg="checked gnum-10000e policy-4294967295, ret-no-match, act-accept"
id=65308 trace_id=4 func=__iprope_check_one_policy line=2131 msg="checked gnum-10000e policy-4294967295, ret-no-match, act-accept"
id=65308 trace_id=4 func=__iprope_check_one_policy line=2131 msg="checked gnum-10000e policy-4294967295, ret-no-match, act-accept"
id=65308 trace_id=4 func=__iprope_check_one_policy line=2131 msg="checked gnum-10000e policy-4294967295, ret-no-match, act-accept"
id=65308 trace_id=4 func=__iprope_check_one_policy line=2131 msg="checked gnum-10000e policy-4294967295, ret-no-match, act-accept"
id=65308 trace_id=4 func=__iprope_check_one_policy line=2131 msg="checked gnum-10000e policy-4294967295, ret-no-match, act-accept"
id=65308 trace_id=4 func=__iprope_check_one_policy line=2131 msg="checked gnum-10000e policy-4294967295, ret-no-match, act-accept"
id=65308 trace_id=4 func=__iprope_check_one_policy line=2131 msg="checked gnum-10000e policy-4294967295, ret-no-match, act-accept"
id=65308 trace_id=4 func=__iprope_check_one_policy line=2131 msg="checked gnum-10000e policy-4294967295, ret-no-match, act-accept"
id=65308 trace_id=4 func=__iprope_check_one_policy line=2131 msg="checked gnum-10000e policy-4294967295, ret-no-match, act-accept"
id=65308 trace_id=4 func=__iprope_check_one_policy line=2131 msg="checked gnum-10000e policy-4294967295, ret-no-match, act-accept"
id=65308 trace_id=4 func=__iprope_check_one_policy line=2131 msg="checked gnum-10000e policy-4294967295, ret-no-match, act-accept"
id=65308 trace_id=4 func=__iprope_check_one_policy line=2131 msg="checked gnum-10000e policy-4294967295, ret-no-match, act-accept"
id=65308 trace_id=4 func=__iprope_check_one_policy line=2131 msg="checked gnum-10000e policy-4294967295, ret-no-match, act-accept"
id=65308 trace_id=4 func=__iprope_check_one_policy line=2131 msg="checked gnum-10000e policy-4294967295, ret-no-match, act-accept"
id=65308 trace_id=4 func=__iprope_check_one_policy line=2131 msg="checked gnum-10000e policy-4294967295, ret-no-match, act-accept"
id=65308 trace_id=4 func=__iprope_check_one_policy line=2131 msg="checked gnum-10000e policy-4294967295, ret-matched, act-accept"
id=65308 trace_id=4 func=__iprope_check_one_policy line=2365 msg="policy-4294967295 is matched, act-drop"
id=65308 trace_id=4 func=__iprope_check line=2412 msg="gnum-10000e check result: ret-matched, act-drop, flag-00000001, flag2-00000000"
id=65308 trace_id=4 func=iprope_policy_group_check line=4892 msg="after check: ret-matched, act-drop, flag-00000001, flag2-00000000"
id=65308 trace_id=4 func=__iprope_check line=2395 msg="gnum-10000f, check-ffffffbffc02c5b4"
id=65308 trace_id=4 func=__iprope_check_one_policy line=2131 msg="checked gnum-10000f policy-4294967295, ret-no-match, act-accept"
id=65308 trace_id=4 func=__iprope_check_one_policy line=2131 msg="checked gnum-10000f policy-4294967295, ret-no-match, act-accept"
id=65308 trace_id=4 func=__iprope_check_one_policy line=2131 msg="checked gnum-10000f policy-4294967295, ret-no-match, act-accept"
id=65308 trace_id=4 func=__iprope_check_one_policy line=2131 msg="checked gnum-10000f policy-4294967295, ret-no-match, act-accept"
id=65308 trace_id=4 func=__iprope_check_one_policy line=2131 msg="checked gnum-10000f policy-4294967295, ret-no-match, act-accept"
id=65308 trace_id=4 func=__iprope_check_one_policy line=2131 msg="checked gnum-10000f policy-4294967295, ret-no-match, act-accept"
id=65308 trace_id=4 func=__iprope_check_one_policy line=2131 msg="checked gnum-10000f policy-4294967295, ret-no-match, act-accept"
id=65308 trace_id=4 func=__iprope_check_one_policy line=2131 msg="checked gnum-10000f policy-4294967295, ret-no-match, act-accept"
id=65308 trace_id=4 func=__iprope_check_one_policy line=2131 msg="checked gnum-10000f policy-4294967295, ret-no-match, act-accept"
id=65308 trace_id=4 func=__iprope_check_one_policy line=2131 msg="checked gnum-10000f policy-4294967295, ret-no-match, act-accept"
id=65308 trace_id=4 func=__iprope_check_one_policy line=2131 msg="checked gnum-10000f policy-4294967295, ret-no-match, act-accept"
id=65308 trace_id=4 func=__iprope_check_one_policy line=2131 msg="checked gnum-10000f policy-4294967295, ret-no-match, act-accept"
id=65308 trace_id=4 func=__iprope_check_one_policy line=2131 msg="checked gnum-10000f policy-4294967295, ret-no-match, act-accept"
id=65308 trace_id=4 func=__iprope_check_one_policy line=2131 msg="checked gnum-10000f policy-4294967295, ret-no-match, act-accept"
id=65308 trace_id=4 func=__iprope_check_one_policy line=2131 msg="checked gnum-10000f policy-4294967295, ret-no-match, act-accept"
id=65308 trace_id=4 func=__iprope_check_one_policy line=2131 msg="checked gnum-10000f policy-4294967295, ret-no-match, act-accept"
id=65308 trace_id=4 func=__iprope_check_one_policy line=2131 msg="checked gnum-10000f policy-4294967295, ret-no-match, act-accept"
id=65308 trace_id=4 func=__iprope_check_one_policy line=2131 msg="checked gnum-10000f policy-4294967295, ret-no-match, act-accept"
id=65308 trace_id=4 func=__iprope_check_one_policy line=2131 msg="checked gnum-10000f policy-4294967295, ret-no-match, act-accept"
id=65308 trace_id=4 func=__iprope_check_one_policy line=2131 msg="checked gnum-10000f policy-4294967295, ret-no-match, act-accept"
id=65308 trace_id=4 func=__iprope_check_one_policy line=2131 msg="checked gnum-10000f policy-4294967295, ret-matched, act-accept"
id=65308 trace_id=4 func=__iprope_check_one_policy line=2365 msg="policy-4294967295 is matched, act-drop"
id=65308 trace_id=4 func=__iprope_check line=2412 msg="gnum-10000f check result: ret-matched, act-drop, flag-00000801, flag2-00000000"
id=65308 trace_id=4 func=iprope_policy_group_check line=4892 msg="after check: ret-matched, act-drop, flag-00000801, flag2-00000000"
id=65308 trace_id=4 func=fw_local_in_handler line=611 msg="iprope_in_check() check failed on policy 0, drop"

Firstly, I can't see the output interface assigned:

id=65308 trace_id=4 func=rpdb_srv_match_input line=1158 msg="Match policy routing id=1: to 172.16.255.2 via ifindex-57"
id=65308 trace_id=4 func=vf_ip_route_input_common line=2612 msg="find a route: flag=84000000 gw-172.16.255.2 via root"
id=65308 trace_id=4 func=iprope_access_proxy_check line=458 msg="in-[vlan11], out-[], skb_flags-02000000, vid-0"

Secondly, I don't understand why act-drop happens (line=2365) if before there is act-accept (line=2131).

If I disable Policy Route and add just static route to 9.9.9.9 via VRF 20, everything works.
But I don't need static route to VRF20, I need to push just certain users.

The problem has been tested with 7.0.15, 7.2.8 and 7.4.4.

Any help is kindly appreciated! Thank you.

[lgupta]

Hello Minotaur, Good day!

Could you also please share the output of routing table for Main VRF?

For policy route to work, there should be always be an active route in the routing table. You can have same AD and higher priority for 0.0.0.0/0 static route for interface "vrf-main-to-20".

I suspect you are missing that, hence it is only working when you add static route.

Thank you!

[Minotaur]

Hello @lgupta,
thank you for reply. I've got your point. The difficulty is that I have default route in VRF 0 pointing to SD-WAN interface, thus I cannot add one more pointing to VRF 20 because "You cannot have duplicated routes on SD-WAN and non-SD-WAN interfaces".
Moreover, if I have static route 9.9.9.9/32 pointing to VRF 20 and policy route enabled simultaneously, it does not work:

gw (root) # get router info routing-table details 9.9.9.9/32

Routing table for VRF=0
Routing entry for 9.9.9.9/32
  Known via "static", distance 10, metric 0, best
  * vrf 0 172.16.255.2, via vrf-main-to-20

gw (root) # show router policy 1
config router policy
    edit 1
        set srcaddr "Server:NS"
        set dst "9.9.9.9/255.255.255.255"
        set gateway 172.16.255.2
        set output-device "vrf-main-to-20"
    next
end

Result is the same:

gw (root) # diag debug enable

gw (root) # diag debug flow show function-name enable
show function name

gw (root) # diag debug flow filter daddr 9.9.9.9

gw (root) # diag debug flow trace start 2

gw (root) # id=65308 trace_id=5 func=print_pkt_detail line=5879 msg="vd-root:0 received a packet(proto=1, 10.1.1.2:39943->9.9.9.9:2048) tun_id=0.0.0.0 from vlan11. type=8, code=0, id=39943, seq=2."
id=65308 trace_id=5 func=init_ip_session_common line=6063 msg="allocate a new session-000bf8e3"
id=65308 trace_id=5 func=rpdb_srv_match_input line=1158 msg="Match policy routing id=1: to 172.16.255.2 via ifindex-57"
id=65308 trace_id=5 func=vf_ip_route_input_common line=2612 msg="find a route: flag=84000000 gw-172.16.255.2 via root"
id=65308 trace_id=5 func=fw_local_in_handler line=611 msg="iprope_in_check() check failed on policy 0, drop"
id=65308 trace_id=6 func=print_pkt_detail line=5879 msg="vd-root:0 received a packet(proto=1, 10.1.1.2:39943->9.9.9.9:2048) tun_id=0.0.0.0 from vlan11. type=8, code=0, id=39943, seq=3."
id=65308 trace_id=6 func=init_ip_session_common line=6063 msg="allocate a new session-000bf8fd"
id=65308 trace_id=6 func=rpdb_srv_match_input line=1158 msg="Match policy routing id=1: to 172.16.255.2 via ifindex-57"
id=65308 trace_id=6 func=vf_ip_route_input_common line=2612 msg="find a route: flag=84000000 gw-172.16.255.2 via root"
id=65308 trace_id=6 func=fw_local_in_handler line=611 msg="iprope_in_check() check failed on policy 0, drop"

[lgupta]

Hello Minotaur, Good day!
Thank you for feedback.

ahh, I just realized you have VLANs under NPU links. Honestly, I would have opened a TAC case so an engineer can live troubleshoot or lab this up IF REQUIRED.

Thanks!

[Minotaur]

The situation is the same with software vlinks.
What I've got from further debugging. There is correct route to 9.9.9.9:

gw (root) # get router info routing-table details 9.9.9.9

Routing table for VRF=0
Routing entry for 9.9.9.9/32
  Known via "static", distance 10, metric 0, best
  * vrf 0 172.16.255.2, via vrf-main-to-20


Routing table for VRF=20
Routing entry for 0.0.0.0/0
  Known via "static", distance 1, metric 0, best
  * vrf 20 XX.XX.136.233, via vlan96-14, origin 2

When policy route from 10.1.1.2 to 9.9.9.9 is disabled then I get correct firewall policy lookup:

gw (root) # diagnose firewall iprope lookup 10.1.1.2 0 9.9.9.9 0 1.8 vlan11 policy
No authentication.
firewall policy id: 37
firewall proxy-policy id: 0
matched policy_type: policy
policy_action: accept

If policy route is enabled then lookup fails:

gw (root) # diagnose firewall iprope lookup 10.1.1.2 0 9.9.9.9 0 1.8 vlan11 policy
No authentication.
No policy matched
sec_default_action: deny
firewall policy id: 0
firewall proxy-policy id: 0

And it looks weird to me.

Indeed, we're going to open a case to TAC.

Thank you.