| Solution |
When the system works as expected, the routing table will look like the following output.
get router info routing-table all
Codes: K - kernel, C - connected, S - static, R - RIP, B - BGP
O - OSPF, IA - OSPF inter area
N1 - OSPF NSSA external type 1, N2 - OSPF NSSA external type 2
E1 - OSPF external type 1, E2 - OSPF external type 2
i - IS-IS, L1 - IS-IS level-1, L2 - IS-IS level-2, ia - IS-IS inter area
* - candidate default
S* 0.0.0.0/0 [10/0] via 192.168.183.254, port2
S 10.0.0.0/8 [10/0] via 192.168.183.254, port2
S 10.176.0.0/23 [10/0] is directly connected, to3600
C 10.177.0.0/23 is directly connected, port1
C 192.168.182.0/23 is directly connected, port2
Ping between PC-A and PC-B, and run the below debug:
diagnose debug flow filter proto 1
diagnose debug flow show function-name enable
diagnose debug console timestamp enable
diagnose debug flow trace start 100
diagnose debug enable
id=20085 trace_id=10 msg="vd-root received a packet(proto=1, 10.177.0.21:512->10.176.0.23:8) from port1."
id=20085 trace_id=10 msg="allocate a new session-0000030b"
id=20085 trace_id=10 msg="find a route: gw-10.176.0.23 via to3600"
id=20085 trace_id=10 msg="Allowed by Policy-2:"
id=20085 trace_id=10 msg="enter IPsec interface-to3600"
id=20085 trace_id=10 msg="encrypted, and send to 192.168.182.225 with source 192.168.183.37"
id=20085 trace_id=10 msg="send to 192.168.182.225 via intf-port2"
id=20085 trace_id=11 msg="vd-root received a packet(proto=1, 10.177.0.21:512->10.176.0.23:8) from port1."
id=20085 trace_id=11 msg="Find an existing session, id-0000030b, original direction"
id=20085 trace_id=11 msg="enter IPsec interface-to3600"
id=20085 trace_id=11 msg="encrypted, and send to 192.168.182.225 with source 192.168.183.37"
id=20085 trace_id=11 msg="send to 192.168.182.225 via intf-port2"
Verify the routing cache:
diagnose ip rtcache list <-- For more info, see Troubleshooting Tip: Route cache is removed from FortiGates running new kernel version.
family=02 tab=254 vf=0 type=01 tos=0 flag=00000200
10.176.0.23@17(to3600)->10.177.0.21@9(port1) gwy=0.0.0.0 prefsrc=0.0.0.0
ci: ref=2 lastused=41 expire=0 err=00000000 used=0 br=0 pmtu=1500
family=02 tab=254 vf=0 type=01 tos=0 flag=00000200
10.177.0.21@9(port1)->10.176.0.23@17(to3600) gwy=0.0.0.0 prefsrc=0.0.0.0
ci: ref=2 lastused=111 expire=0 err=00000000 used=0 br=0 pmtu=1436
Check the session list:
diagnose sys session list
session info: proto=1 proto_state=00 expire=55 timeout=3600 flags=00000000 sockflag=00000000 sockport=0 av_idx=0 use=3
bandwidth=0/sec guaranteed_bandwidth=0/sec traffic=0/sec prio=0 ha_id=0 hakey=44
tunnel=/
state=may_dirty npu
statistic(bytes/packets/err): org=60/1/0 reply=60/1/0 tuples=2
orgin->sink: org pre->post, reply pre->post dev=9->17/17->9 gwy=10.176.0.23/10.177.0.21
hook=pre dir=org act=noop 10.177.0.21:512->10.176.0.23:8(0.0.0.0:0)
hook=post dir=reply act=noop 10.176.0.23:512->10.177.0.21:0(0.0.0.0:0)
misc=0 policy_id=2 auth_info=0 ids=0xf6f91e58 vd=0 serial=0000034c tos=ff/ff app=0
Compare to the output when the tunnel is down.
Routing table
get router info routing-table all
Codes: K - kernel, C - connected, S - static, R - RIP, B - BGP
O - OSPF, IA - OSPF inter area
N1 - OSPF NSSA external type 1, N2 - OSPF NSSA external type 2
E1 - OSPF external type 1, E2 - OSPF external type 2
i - IS-IS, L1 - IS-IS level-1, L2 - IS-IS level-2, ia - IS-IS inter area
* - candidate default
S* 0.0.0.0/0 [10/0] via 192.168.183.254, port2
S 10.0.0.0/8 [10/0] via 192.168.183.254, port2
C 10.177.0.0/23 is directly connected, port1
C 192.168.182.0/23 is directly connected, port2
Ping from PC-A to PC-B:
diagnose debug flow filter proto 1
diagnose debug flow show function-name enable
diag deb console timestamp enable
diagnose debug flow trace start 100
diagnose debug enable
id=20085 trace_id=12 msg="vd-root received a packet(proto=1, 10.177.0.21:512->10.176.0.23:8) from port1."
id=20085 trace_id=12 msg="allocate a new session-00000433"
id=20085 trace_id=12 msg="find a route: gw-192.168.183.254 via port2"
id=20085 trace_id=12 msg="find SNAT: IP-192.168.183.37, port-55933"
id=20085 trace_id=12 msg="Allowed by Policy-1: SNAT"
id=20085 trace_id=12 msg="SNAT 10.177.0.21->192.168.183.37:55933"
id=20085 trace_id=13 msg="vd-root received a packet(proto=1, 10.177.0.21:512->10.176.0.23:8) from port1."
id=20085 trace_id=13 msg="Find an existing session, id-00000433, original direction"
id=20085 trace_id=13 msg="SNAT 10.177.0.21->192.168.183.37:55933"
This is also normal as there is another route for the 10.0.0.0/8 network using Port2. If the tunnel comes back and some applications keep trying to reach PC-B, then the traffic will not flow correctly.
get router info routing-table all
Codes: K - kernel, C - connected, S - static, R - RIP, B - BGP
O - OSPF, IA - OSPF inter area
N1 - OSPF NSSA external type 1, N2 - OSPF NSSA external type 2
E1 - OSPF external type 1, E2 - OSPF external type 2
i - IS-IS, L1 - IS-IS level-1, L2 - IS-IS level-2, ia - IS-IS inter area
* - candidate default
S* 0.0.0.0/0 [10/0] via 192.168.183.254, port2
S 10.0.0.0/8 [10/0] via 192.168.183.254, port2
S 10.176.0.0/23 [10/0] is directly connected, to3600
C 10.177.0.0/23 is directly connected, port1
C 192.168.182.0/23 is directly connected, port2
id=20085 trace_id=14 msg="vd-root received a packet(proto=1, 10.177.0.21:512->10.176.0.23:8) from port1."
id=20085 trace_id=14 msg="Find an existing session, id-00000433, original direction"
id=20085 trace_id=14 msg="find a route: gw-192.168.183.254 via port2"
id=20085 trace_id=14 msg="find SNAT: IP-192.168.183.37, port-55936"
id=20085 trace_id=14 msg="SNAT 10.177.0.21->192.168.183.37:55933"
id=20085 trace_id=15 msg="vd-root received a packet(proto=1, 10.177.0.21:512->10.176.0.23:8) from port1."
id=20085 trace_id=15 msg="Find an existing session, id-00000433, original direction"
id=20085 trace_id=15 msg="SNAT 10.177.0.21->192.168.183.37:55933"
Traffic uses Port2, but not the IPSEC interface. This is because the route that was created when the IPSEC tunnel was down was still active.
family=02 tab=254 vf=0 type=01 tos=0 flag=00000200
10.177.0.21@9(port1)->10.176.0.23@8(port2) gwy=192.168.183.254 prefsrc=0.0.0.0
ci: ref=2 lastused=94 expire=0 err=00000000 used=0 br=0 pmtu=1500
The solution here is to use a firewall policy. Add a deny policy for traffic using the Public Interface instead of the IPSEC tunnel.
For example:
config firewall policy
edit 3
set srcintf "port1"
set dstintf "port2"
set srcaddr "all"
set dstaddr "RemoteLAN"
set schedule "always"
set service "ALL"
next
edit 1
id=20085 trace_id=20 msg="vd-root received a packet(proto=1, 10.177.0.21:512->10.176.0.23:8) from port1."
id=20085 trace_id=20 msg="allocate a new session-00000538"
id=20085 trace_id=20 msg="find a route: gw-192.168.183.254 via port2"
id=20085 trace_id=20 msg="Denied by forward policy check"
id=20085 trace_id=21 msg="vd-root received a packet(proto=1, 10.177.0.21:512->10.176.0.23:8) from port1."
id=20085 trace_id=21 msg="allocate a new session-0000053b"
id=20085 trace_id=21 msg="find a route: gw-192.168.183.254 via port2"
id=20085 trace_id=21 msg="Denied by forward policy check"
Use a black hole route for a network destined over a VPN tunnel with a higher cost. This causes the traffic to be dropped when it hits the blackhole route.
get router info routing-table all
Codes: K - kernel, C - connected, S - static, R - RIP, B - BGP
O - OSPF, IA - OSPF inter area
N1 - OSPF NSSA external type 1, N2 - OSPF NSSA external type 2
E1 - OSPF external type 1, E2 - OSPF external type 2
i - IS-IS, L1 - IS-IS level-1, L2 - IS-IS level-2, ia - IS-IS inter area
* - candidate default
S* 0.0.0.0/0 [10/0] via 192.168.183.254, port2
S 10.0.0.0/8 [10/0] via 192.168.183.254, port2
S 10.176.0.0/23 [20/0] is a summary, Null
C 10.177.0.0/23 is directly connected, port1
C 192.168.182.0/23 is directly connected, port2
config router static
edit 1
set device "port2"
set gateway 192.168.183.254
next
edit 2
set device "to3600"
set dst 10.176.0.0 255.255.254.0
next
edit 3
set device "port2"
set dst 10.0.0.0 255.0.0.0
set gateway 192.168.183.254
next
edit 4
set blackhole enable
set distance 20
set dst 10.176.0.0 255.255.254.0
next
end
diagnose debug flow trace start 2
id=20085 trace_id=22 msg="vd-root received a packet(proto=1, 10.177.0.21:512->10.176.0.23:8) from port1." id=20085 trace_id=22 msg="allocate a new session-0000059a" id=20085 trace_id=23 msg="vd-root received a packet(proto=1, 10.177.0.21:512->10.176.0.23:8) from port1." id=20085 trace_id=23 msg="allocate a new session-0000059c"
When the tunnel comes back again, it starts using the route to the IPsec interface, not the NULL route (blackhole route).
get router info routing-table database
Codes: K - kernel, C - connected, S - static, R - RIP, B - BGP O - OSPF, IA - OSPF inter area N1 - OSPF NSSA external type 1, N2 - OSPF NSSA external type 2 E1 - OSPF external type 1, E2 - OSPF external type 2 i - IS-IS, L1 - IS-IS level-1, L2 - IS-IS level-2, ia - IS-IS inter area > - selected route, * - FIB route, p - stale info
S *> 0.0.0.0/0 [10/0] via 192.168.183.254, port2 S *> 10.0.0.0/8 [10/0] via 192.168.183.254, port2 S 10.176.0.0/23 [20/0] is a summary, Null S *> 10.176.0.0/23 [10/0] is directly connected, to3600 C *> 10.177.0.0/23 is directly connected, port1 C *> 192.168.182.0/23 is directly connected, port2
|
