Fortinet 1000D Routing issue

[maxblaze]

Hi Guys,

I have this weird issue where i have routing on this specific network via BGP but is not reachable (yes IP is reachable  outside of my network) when i conduct trace route probe is not able to get out. (see screenshot) 

I even have default route 0.0.0.0 0.0.0.0 x.x.x.x

and also BGP route, but i seems i cant get out on my network.

This happens for this several specific ip now.

Yes all my Upstream link is added on the outgoing interface so i don't think it is ipv4  policy issue any idea?

[umutkacar]

Hi,

Can you share the outputs of following, when you're trying to reach that IP?

diagnose sniffer packet any "host <sourceIP> and host <destinationIP>" 4 a

if this doesn't show anything, go with:

diagnose sniffer packet any "host <sourceIP> or host <destinationIP>" 4 a

and finally:

diag debug reset diag debug flow filter add <destinationIP> diagnose debug flow show function-name enable diag debug flow trace start 100 diag debug enable

Note: you should replace <source/destinationIP> parts with your IP, i.e. 192.168.1.1

[maxblaze]

Hi Umut,

First i would like to thank you for giving me chance to solve this issue.

diagnose sniffer packet any "host 121.127.X.X  or host 202.75.X.X" 4 a

filters=[host 121.127.X.X or host 202.75.X.X] 0.997330 port9 in 216.58.221.237.443 -> 121.127.X.X.64647: syn 3705287683 ack 3219269829 0.997330 port9 in 203.77.191.30.443 -> 121.127.X.X.57464: ack 1211661800 0.997330 port9 out 121.127.X.X.64647 -> 216.58.221.237.443: ack 3705287684 1.017330 port9 in 103.252.200.61.48043 -> 121.127.X.X.27024: syn 4281788123 1.017330 port9 in 52.77.149.91.443 -> 121.127.X.X.42554: syn 2310957938 ack 3449684641 1.027330 port9 in 54.239.20.230.443 -> 121.127.X.X.62161: psh 2105668991 ack 4243965703 1.027330 port9 in 54.239.20.230.443 -> 121.127.X.X.62161: fin 2105669044 ack 4243965703 1.027330 port9 out 121.127.X.X.62161 -> 54.239.20.230.443: rst 4243965703 ack 2105669044 1.027330 port9 out 121.127.X.X.42554 -> 52.77.149.91.443: ack 2310957939 1.047330 port9 in 72.21.214.67.443 -> 121.127.X.X.62155: ack 589007303 1.047330 port9 in 72.21.214.67.443 -> 121.127.X.X.62176: syn 713846659 ack 3456979986 1.057330 port9 out 121.127.X.X.62176 -> 72.21.214.67.443: ack 713846660 1.057330 port9 in 36.80.245.54.1025 -> 121.127.X.X.61982: udp 20 1.067330 port9 in 198.11.132.53.80 -> 121.127.X.X.48025: fin 2229694763 ack 3332770607 1.067330 port9 out 121.127.X.X.55152 -> 17.253.87.207.443: fin 1878731619 ack 2253669542 1.077330 port9 out 121.127.X.X.57464 -> 203.77.191.30.443: ack 319312436 1.087330 port9 in 17.253.87.207.443 -> 121.127.X.X.55152: fin 2253669566 ack 1878731619 1.097330 port9 out 121.127.X.X.55152 -> 17.253.87.207.443: rst 1878731619 1.097330 port9 out 121.127.X.X.55152 -> 17.253.87.207.443: rst 1878731619 1.097330 port9 out 121.127.X.X.55152 -> 17.253.87.207.443: rst 1878731620 1.117330 port9 out 121.127.X.X -> 66.151.55.110: icmp: time exceeded in-transit 1.137330 port9 in 110.54.161.100.35578 -> 121.127.X.X.6881: syn 2091729113 1.147330 port9 out 121.127.X.X.38992 -> 175.136.87.48.39350: udp 20 1.147330 port9 out 121.127.X.X -> 189.229.5.149: icmp: net 36.255.107.200 unreachable 1.157330 port9 in 157.240.179.129.443 -> 121.127.X.X.47588: psh fin 2009620164 ack 794792065 1.177330 port9 out 121.127.X.X.38992 -> 112.199.208.157.35187: udp 52 1.187330 port9 in 3.208.52.10.443 -> 121.127.X.X.55314: fin 3508265103 ack 3745769980 1.197330 port9 out 121.127.X.X.55314 -> 3.208.52.10.443: rst 3745769980 1.197330 port9 out 121.127.X.X.55314 -> 3.208.52.10.443: rst 3745769981 1.197330 port9 in 177.33.57.31.57229 -> 121.127.X.X.27024: syn 4098565558 1.197330 port9 out 121.127.X.X.55314 -> 3.208.52.10.443: rst 3745769980 1.197330 port9 out 121.127.X.X.64647 -> 216.58.221.237.443: ack 3705291430 1.197330 port9 in 216.58.221.237.443 -> 121.127.X.X.64647: psh 3705298170 ack 3219271379 1.207330 port9 in 64.233.189.189.443 -> 121.127.X.X.58547: udp 40

diag debug reset diag debug flow filter add <destinationIP> diagnose debug flow show function-name enable diag debug flow trace start 100 diag debug enable

PING 202.75.X.X (202.75.X.X): 56 data bytes id=20085 trace_id=1 func=print_pkt_detail line=4793 msg="vd-root received a packet(proto=1, 121.127.X.X:39680->202.75.X.X:2048) from local. type=8, code=0, id=39680, seq=0." id=20085 trace_id=1 func=init_ip_session_common line=4944 msg="allocate a new session-5f30ac8b" id=20085 trace_id=2 func=print_pkt_detail line=4793 msg="vd-root received a packet(proto=1, 121.127.X.X:39680->202.75.X.X:2048) from local. type=8, code=0, id=39680, seq=1." id=20085 trace_id=2 func=resolve_ip_tuple_fast line=4857 msg="Find an existing session, id-5f30ac8b, original direction" id=20085 trace_id=3 func=print_pkt_detail line=4793 msg="vd-root received a packet(proto=1, 121.127.X.X:39680->202.75.X.X:2048) from local. type=8, code=0, id=39680, seq=2." id=20085 trace_id=3 func=resolve_ip_tuple_fast line=4857 msg="Find an existing session, id-5f30ac8b, original direction" id=20085 trace_id=4 func=print_pkt_detail line=4793 msg="vd-root received a packet(proto=1, 121.127.X.X:39680->202.75.X.X:2048) from local. type=8, code=0, id=39680, seq=3." id=20085 trace_id=4 func=resolve_ip_tuple_fast line=4857 msg="Find an existing session, id-5f30ac8b, original direction" id=20085 trace_id=5 func=print_pkt_detail line=4793 msg="vd-root received a packet(proto=1, 121.127.X.X:39680->202.75.X.X:2048) from local. type=8, code=0, id=39680, seq=4." id=20085 trace_id=5 func=resolve_ip_tuple_fast line=4857 msg="Find an existing session, id-5f30ac8b, original direction"

--- 202.75.X.X ping statistics --- 5 packets transmitted, 0 packets received, 100% packet loss

*Sniff when static route is removed and ip become unreachable and cant get out on the first hop despite having bgp and default route.

Again, Thank you,

max

[maxblaze]

*Sniff when static route is enable and ip become reachable.

filters=[host 121.127.X.X or host 202.75.X.X] 0.695892 port9 in 54.69.14.161.80 -> 121.127.X.X.59631: syn 598280286 ack 2755532838 0.695892 port9 out 121.127.X.X.59631 -> 54.69.14.161.80: ack 598280287 0.695892 port9 out 121.127.X.X.59418 -> 54.212.249.221.443: fin 3937115006 ack 2402430863 0.695892 port9 out 121.127.X.X.59372 -> 54.254.217.108.443: 3866316845 ack 3407335597 0.695892 port9 out 121.127.X.X.59372 -> 54.254.217.108.443: psh 3866318293 ack 3407335597 0.695892 port9 out 121.127.X.X.34405 -> 23.198.113.166.80: ack 2626978106 0.695892 port9 out 121.127.X.X.34405 -> 23.198.113.166.80: ack 2626981002 0.695892 port9 out 121.127.X.X.39665 -> 23.198.113.166.80: ack 164740231 0.705892 port9 in 18.136.208.233.443 -> 121.127.X.X.38780: fin 3653434923 ack 3640707743 0.715892 port9 in 157.240.15.16.443 -> 121.127.X.X.56691: udp 33 0.715892 port9 in 86.41.161.124.40959 -> 121.127.X.X.6881: udp 20 0.715892 port9 out 121.127.X.X.6881 -> 86.41.161.124.40959: udp 1386 0.715892 port9 out 121.127.X.X.6881 -> 86.41.161.124.40959: udp 1386 0.725892 port9 in 103.10.226.17.55525 -> 121.127.X.X.6881: udp 20 0.725892 port9 out 121.127.X.X.6881 -> 103.10.226.17.55525: udp 1386 0.725892 port9 in 8.8.8.8.53 -> 121.127.X.X.55102: udp 82 0.725892 port9 in 8.8.8.8.53 -> 121.127.X.X.16641: udp 90 0.725892 port9 out 121.127.X.X.62088 -> 116.93.47.35.443: ack 440893833 0.725892 port9 in 103.10.226.17.55525 -> 121.127.X.X.6881: udp 20 0.735892 port9 in 111.91.8.93.16798 -> 121.127.X.X.35095: syn 2261963743 0.735892 port9 out 121.127.X.X.45446 -> 52.74.215.76.443: syn 350069776 0.745892 port9 in 5.65.228.216.8999 -> 121.127.X.X.50976: ack 3922025095 0.745892 port9 in 18.136.117.61.10300 -> 121.127.X.X.58402: psh 1312057176 ack 3008374177 0.755892 port9 in 132.255.240.158.25792 -> 121.127.X.X.6881: udp 20 0.755892 port9 in 196.188.176.135.27727 -> 121.127.X.X.61982: syn 1113599380 0.765892 port9 in 54.254.217.108.443 -> 121.127.X.X.59372: ack 3866319394 0.765892 port9 in 8.8.8.8.53 -> 121.127.X.X.65165: udp 99 0.765892 port9 out 121.127.X.X.51935 -> 52.85.153.3.443: 2491569118 ack 4023467520 0.765892 port9 in 173.32.220.158.30732 -> 121.127.X.X.6881: udp 20 0.765892 port9 out 121.127.X.X.6881 -> 173.32.220.158.30732: udp 1386 0.765892 port9 out 121.127.X.X.43826 -> 52.222.255.93.80: syn 2098093425 0.785892 port9 in 173.243.138.103.80 -> 121.127.X.X.64287: 4051470833 ack 1661166343 0.785892 port9 in 173.243.138.103.80 -> 121.127.X.X.64287: 4051472293 ack 1661166343 0.785892 port9 in 173.243.138.103.80 -> 121.127.X.X.64287: 4051473753 ack 1661166343 0.785892 port9 in 173.243.138.103.80 -> 121.127.X.X.64287: 4051475213 ack 1661166343 0.785892 port9 in 173.243.138.103.80 -> 121.127.X.X.64287: 4051476673 ack 1661166343 0.785892 port9 in 173.243.138.103.80 -> 121.127.X.X.64287: 4051478133 ack 1661166343 0.785892 port9 in 173.243.138.103.80 -> 121.127.X.X.64287: 4051479593 ack 1661166343 0.785892 port9 in 173.243.138.103.80 -> 121.127.X.X.64287: 4051481053 ack 1661166343 0.785892 port9 out 121.127.X.X.64287 -> 173.243.138.103.80: ack 4051473753 0.785892 port9 in 173.243.138.103.80 -> 121.127.X.X.64287: 4051482513 ack 1661166343 0.785892 port9 in 173.243.138.103.80 -> 121.127.X.X.64287: psh fin 4051483973 ack 1661166343 0.785892 port9 out 121.127.X.X.64287 -> 173.243.138.103.80: ack 4051476673 0.785892 port9 out 121.127.X.X.64287 -> 173.243.138.103.80: ack 4051479593 0.785892 port9 out 121.127.X.X.64287 -> 173.243.138.103.80: ack 4051482513 0.785892 port9 out 121.127.X.X.64287 -> 173.243.138.103.80: ack 4051485022 0.785892 port9 out 121.127.X.X -> 172.105.217.71: icmp: time exceeded in-transit 0.785892 port9 out 121.127.X.X.59632 -> 52.192.163.135.80: syn 773240971 0.785892 port9 out 121.127.X.X.64287 -> 173.243.138.103.80: ack 4051485022 0.795892 port9 in 52.85.153.3.443 -> 121.127.X.X.51935: ack 2491569119 0.795892 port9 in 52.74.215.76.443 -> 121.127.X.X.45446: syn 158939023 ack 350069777 0.805892 port9 out 121.127.X.X.45446 -> 52.74.215.76.443: ack 158939024 0.805892 port9 out 121.127.X.X.58402 -> 18.136.117.61.10300: ack 1312057760 0.805892 port9 in 108.177.97.114.80 -> 121.127.X.X.36694: fin 1458926954 ack 1762227318 0.805892 port9 in 52.222.255.93.80 -> 121.127.X.X.43826: syn 2649158359 ack 2098093426 0.805892 port9 out 121.127.X.X.43826 -> 52.222.255.93.80: ack 2649158360 0.815892 port9 out 121.127.X.X.56750 -> 107.155.58.101.80: syn 3320904347 0.815892 port9 in 119.161.14.17.443 -> 121.127.X.X.60866: psh 3970531083 ack 419163389 0.825892 port9 out 121.127.X.X.45000 -> 205.185.208.142.443: rst 215922996 0.825892 port9 out 121.127.X.X.45035 -> 205.185.208.142.443: ack 1382752182 0.835892 port9 out 121.127.X.X.52726 -> 190.199.166.254.57371: udp 214 0.835892 port9 out 121.127.X.X.63891 -> 41.139.248.6.18318: syn 2993650667 0.845892 port9 out 121.127.X.X.47896 -> 157.240.15.16.443: psh 170741646 ack 873571228 0.855892 port9 in 103.10.226.17.55525 -> 121.127.X.X.6881: udp 20 0.855892 port9 out 121.127.X.X.6881 -> 103.10.226.17.55525: udp 1252

PING 202.75.X.X (202.75.X.X): 56 data bytes id=20085 trace_id=11 func=print_pkt_detail line=4793 msg="vd-root received a packet(proto=1, 121.127.X.X:40192->202.75.X.X:2048) from local. type=8, code=0, id=40192, seq=0." id=20085 trace_id=11 func=init_ip_session_common line=4944 msg="allocate a new session-5f3d3db1" 64 bytes from 202.75.X.X: icmp_seq=0 ttl=118 time=70.0 ms id=20085 trace_id=12 func=print_pkt_detail line=4793 msg="vd-root received a packet(proto=1, 202.75.X.X:40192->121.127.X.X:0) from port9. type=0, code=0, id=40192, seq=0." id=20085 trace_id=12 func=resolve_ip_tuple_fast line=4857 msg="Find an existing session, id-5f3d3db1, reply direction" id=20085 trace_id=12 func=vf_ip_route_input_common line=2586 msg="find a route: flag=80000000 gw-121.127.X.X via root" id=20085 trace_id=13 func=print_pkt_detail line=4793 msg="vd-root received a packet(proto=1, 121.127.X.X:40192->202.75.X.X:2048) from local. type=8, code=0, id=40192, seq=1." id=20085 trace_id=13 func=resolve_ip_tuple_fast line=4857 msg="Find an existing session, id-5f3d3db1, original direction" 64 bytes from 202.75.X.X: icmp_seq=1 ttl=118 time=70.0 ms id=20085 trace_id=14 func=print_pkt_detail line=4793 msg="vd-root received a packet(proto=1, 202.75.X.X:40192->121.127.X.X:0) from port9. type=0, code=0, id=40192, seq=1." id=20085 trace_id=14 func=resolve_ip_tuple_fast line=4857 msg="Find an existing session, id-5f3d3db1, reply direction" id=20085 trace_id=14 func=vf_ip_route_input_common line=2586 msg="find a route: flag=80000000 gw-121.127.X.X via root" id=20085 trace_id=15 func=print_pkt_detail line=4793 msg="vd-root received a packet(proto=1, 121.127.X.X:40192->202.75.X.X:2048) from local. type=8, code=0, id=40192, seq=2." id=20085 trace_id=15 func=resolve_ip_tuple_fast line=4857 msg="Find an existing session, id-5f3d3db1, original direction" 64 bytes from 202.75.X.X: icmp_seq=2 ttl=118 time=70.0 ms id=20085 trace_id=16 func=print_pkt_detail line=4793 msg="vd-root received a packet(proto=1, 202.75.X.X:40192->121.127.X.X:0) from port9. type=0, code=0, id=40192, seq=2." id=20085 trace_id=16 func=resolve_ip_tuple_fast line=4857 msg="Find an existing session, id-5f3d3db1, reply direction" id=20085 trace_id=16 func=vf_ip_route_input_common line=2586 msg="find a route: flag=80000000 gw-121.127.X.X via root" id=20085 trace_id=17 func=print_pkt_detail line=4793 msg="vd-root received a packet(proto=1, 121.127.X.X:40192->202.75.X.X:2048) from local. type=8, code=0, id=40192, seq=3." id=20085 trace_id=17 func=resolve_ip_tuple_fast line=4857 msg="Find an existing session, id-5f3d3db1, original direction" 64 bytes from 202.75.X.X: icmp_seq=3 ttl=118 time=70.0 ms id=20085 trace_id=18 func=print_pkt_detail line=4793 msg="vd-root received a packet(proto=1, 202.75.X.X:40192->121.127.X.X:0) from port9. type=0, code=0, id=40192, seq=3." id=20085 trace_id=18 func=resolve_ip_tuple_fast line=4857 msg="Find an existing session, id-5f3d3db1, reply direction" id=20085 trace_id=18 func=vf_ip_route_input_common line=2586 msg="find a route: flag=80000000 gw-121.127.X.X via root" id=20085 trace_id=19 func=print_pkt_detail line=4793 msg="vd-root received a packet(proto=1, 121.127.X.X:40192->202.75.X.X:2048) from local. type=8, code=0, id=40192, seq=4." id=20085 trace_id=19 func=resolve_ip_tuple_fast line=4857 msg="Find an existing session, id-5f3d3db1, original direction" 64 bytes from 202.75.X.X: icmp_seq=4 ttl=118 time=70.0 ms