Troubleshooting Tip: FortiOS routing (RIP, OSPF, BGP, static routes, ECMP)

rmetzger · ‎10-19-2009

Description

This article provides a series of initial troubleshooting procedures and diagnostic commands related to FortiOS routing.

The information gathered can be passed to Fortinet Technical Support engineer when opening a support ticket.

Please note that all CLI commands provided below are per VDOM based; if VDOMs are enabled, type first:

config vdom
edit "<my_vdom>"

Scope
FortiGate unit or VDOM in NAT mode
Solution

Content

1. General ECMP information
2. Static routing
3. Recommended procedure to troubleshoot RIP
4. Recommended procedure to troubleshoot OSPF
5. Recommended procedure to troubleshoot BGP

1. General ECMP information

General ECMP information is available in the related article: Technical Note: Configuring link redundancy - traffic load-balancing - ECMP (Equal Cost Multiple Path) - Dual Internet or WAN scenario.

2. Static routing

Information about static routing is available in the related article: Technical Note: Conditions to get a route in the FortiGate routing table (valid next-hop for DHCP, PPPoE, or static routes).

3. Recommended procedure to troubleshoot RIP

3.1 Looking for RIP routes in the routing table:

FGT2 # get router info routing-table all

or use a RIP filter:

FGT2 # get router info routing-table rip

Example output

FGT2 # get router info routing-table all

Codes: K - kernel, C - connected, S - static, R - RIP, B - BGP
O - OSPF, IA - OSPF inter area
N1 - OSPF NSSA external type 1, N2 - OSPF NSSA external type 2
E1 - OSPF external type 1, E2 - OSPF external type 2
i - IS-IS, L1 - IS-IS level-1, L2 - IS-IS level-2, ia - IS-IS inter area
* - candidate default

S* 0.0.0.0/0 [10/0] via 192.168.183.254, wan1
S 10.0.0.0/24 [10/0] is directly connected, ssl.root
R 10.0.0.1/32 [120/2] via 192.168.182.110, wan1, 00:34:52
R 10.0.0.2/32 [120/2] via 192.168.182.110, wan1, 00:34:52
C 10.160.0.0/23 is directly connected, internal
R 10.161.0.0/23 [120/2] via 192.168.182.58, wan1, 00:35:27
S 30.0.0.0/8 [10/0] is directly connected, ssl.root
R 172.16.30.0/24 [120/2] via 192.168.182.110, wan1, 00:34:52
C 192.168.182.0/23 is directly connected, wan1

3.2 - Verify L2 (arp entry) and L3 connectivity between routers

FGT # get system arp

Example output

Address Age(min) Hardware Addr Interface
10.1.1.3 0 00:09:0f:30:29:e4 port1
192.168.183.254 0 00:09:0f:09:32:04 port1

FGT# execute ping <neighbor>

Example output

FGT # execute ping 10.160.0.110

PING 10.160.0.110 (10.160.0.110): 56 data bytes
64 bytes from 10.160.0.110: icmp_seq=0 ttl=255 time=0.1 ms
64 bytes from 10.160.0.110: icmp_seq=1 ttl=255 time=0.0 ms

--- 10.160.0.110 ping statistics ---
2 packets transmitted, 2 packets received, 0% packet loss
round-trip min/avg/max = 0.0/0.0/0.1 ms

3.3 Verify the global and interface RIP status

FGT # get router info protocols

Example output

Routing Protocol is "rip"
Sending updates every 30 seconds with +/-50%, next due in 19 seconds
Timeout after 180 seconds, garbage collect after 120 seconds
Outgoing update filter list for all interface is not set
Incoming update filter list for all interface is not set
Default redistribution metric is 1
Redistributing: connected
Default version control: send version 2, receive version 2
Interface Send Recv Key-chain
internal 2 2
Routing for Networks:
10.0.0.0/8
Routing Information Sources:
Gateway Distance Last Update Bad Packets Bad Routes
10.160.0.110 120 00:00:18 0 5
Distance: (default is 120)

FGT2 # get router info rip interface

Example output

w an1 is up, line protocol is up
RIP is not enabled on this interface
internal is up, line protocol is up
Routing Protocol: RIP
Receive RIP packets
Send RIP packets
Passive interface: Disabled
Split horizon: Enabled with Poisoned Reversed
IP interface address:
10.160.0.106/23
modem is down, line protocol is down
RIP is not enabled on this interface
root is up, line protocol is up
RIP is not enabled on this interface
ssl.root is up, line protocol is up
RIP is not enabled on this interface

3.4 Take a sniffer trace to look for RIP1 or RIP2 packets:

RIPv2 example:

FGT2 # diagnose sniffer packet any "port 520" 4

Example output

interfaces=[any]
filters=[port 520]
9.774426 wan1 in 192.168.182.58.520 -> 224.0.0.9.520: udp 164
14.179910 wan1 in 192.168.182.110.520 -> 224.0.0.9.520: udp 164
29.690209 wan1 out 192.168.182.106.520 -> 224.0.0.9.520: udp 164
35.779359 wan1 in 192.168.182.58.520 -> 224.0.0.9.520: udp 164
37.173912 wan1 in 192.168.182.110.520 -> 224.0.0.9.520: udp 164
58.690207 wan1 out 192.168.182.106.520 -> 224.0.0.9.520: udp 164
70.782579 wan1 in 192.168.182.58.520 -> 224.0.0.9.520: udp 164
77.163482 wan1 in 192.168.182.110.520 -> 224.0.0.9.520: udp 164

3.5   Further troubleshooting can be performed by looking at each packet received, sent, and processed, with the the following command:

FGT# diagnose ip router rip all (enable|disable)
FGT# diagnose debug enable

This command may generate some extensive output; it is also possible to use more specific debug filters instead of "all" to reduce the verbosity.

Example output

RIP: RECV[internal]: Receive from 10.160.0.205:520
RIP: RECV[internal]: RESPONSE version 2 packet size 24
RIP: 0.0.0.0/0 -> 0.0.0.0 family 2 tag 0 metric 1
RIP: RECV[internal]: Receive from 10.160.0.205:520

4. Recommended procedure to troubleshoot OSPF

4.1 Check the routing table with OSPF routes

FGT # get router info routing-table all

or filter OSPF entries

FGT # get router info routing-table ospf

Example output

FGT2 # get router info routing-table all

Codes: K - kernel, C - connected, S - static, R - RIP, B - BGP
O - OSPF, IA - OSPF inter area
N1 - OSPF NSSA external type 1, N2 - OSPF NSSA external type 2
E1 - OSPF external type 1, E2 - OSPF external type 2
i - IS-IS, L1 - IS-IS level-1, L2 - IS-IS level-2, ia - IS-IS inter area
* - candidate default

S* 0.0.0.0/0 [10/0] via 192.168.183.254, wan1
S 10.0.0.0/24 [10/0] is directly connected, ssl.root
O 10.0.0.1/32 [110/100] via 10.160.0.110, internal, 00:06:21
O 10.0.0.2/32 [110/200] via 10.160.0.110, internal, 00:06:21

4.2 Verify the OSPF Neighbors State

FGT2# get router info ospf neighbor

Example output

OSPF process 0:

Neighbor ID Pri State Dead Time Address Interface

10.0.0.23 1 Full/DR 00:00:35 10.1.1.3 port1
10.0.0.25 1 Full/DR 00:00:30 10.142.0.155 port8

4.3    If no neighbors are seen, verify L2 (arp entry) and L3 connectivity between neighbors, and take a sniffer trace:

FGT # get system arp

Example output

Address Age(min) Hardware Addr Interface
10.1.1.3 0 00:09:0f:30:29:e4 port1
192.168.183.254 0 00:09:0f:09:32:04 port1

FGT# execute ping <neighbor>

Example output

FGT# execute ping 10.160.0.110

PING 10.160.0.110 (10.160.0.110): 56 data bytes
64 bytes from 10.160.0.110: icmp_seq=0 ttl=255 time=0.1 ms
64 bytes from 10.160.0.110: icmp_seq=1 ttl=255 time=0.0 ms

--- 10.160.0.110 ping statistics ---
2 packets transmitted, 2 packets received, 0% packet loss
round-trip min/avg/max = 0.0/0.0/0.1 ms

A sniffer trace will tell if OSPF HELLO packets are sent/received :

FGT2# diagnose sniffer packet any "proto 89" 4 (stop sniffer with CTRL+C)

Example output

interfaces=[any]
filters=[proto 89]
0.748972 internal out 10.160.0.106 -> 224.0.0.5: ip-proto-89 48
4.796492 internal in 10.161.0.58 -> 224.0.0.5: ip-proto-89 44

4.4     If Neighbor States are not Full or 2-Ways, check OSPF global state and interface state

If the subnet has more than 2 Neighbors, it is expected to get 2 of them in Full state (DR and BDR) and all the other ones in 2way state (DRother).

FG1# get router info ospf status

Example output

Routing Process "ospf 0" with ID 192.168.182.110
Process uptime is 11 minutes
Process bound to VRF default
Conforms to RFC2328, and RFC1583Compatibility flag is disabled
Supports only single TOS(TOS0) routes
Supports opaque LSA
Do not support Restarting
SPF schedule delay 5 secs, Hold time between two SPFs 10 secs
Refresh timer 10 secs
Number of incomming current DD exchange neighbors 0/5
Number of outgoing current DD exchange neighbors 0/5
Number of external LSA 0. Checksum 0x000000
Number of opaque AS LSA 0. Checksum 0x000000
Number of non-default external LSA 0
External LSA database is unlimited.
Number of LSA originated 1
Number of LSA received 0
Number of areas attached to this router: 1
Area 0.0.0.0 (BACKBONE)
Number of interfaces in this area is 2(3)
Number of fully adjacent neighbors in this area is 0
Area has no authentication
SPF algorithm last executed 00:11:18.710 ago
SPF algorithm executed 2 times
Number of LSA 1. Checksum 0x0008dc
Area 0.0.0.1 (Stub) (Inactive)
Number of interfaces in this area is 0(0)
Number of fully adjacent neighbors in this area is 0
Number of fully adjacent virtual neighbors through this area is 0
Area has no authentication
SPF algorithm executed 0 times
Number of LSA 1. Checksum 0x002028
Shortcutting mode: Disabled, S-bit consensus: no

FGT # get router info ospf interface

Example output

port1 is up, line protocol is up
Internet Address 10.160.0.110/23, Area 0.0.0.0, MTU 1500
Process ID 0, Router ID 192.168.182.110, Network Type BROADCAST, Cost: 10
Transmit Delay is 1 sec, State DR, Priority 1
Designated Router (ID) 192.168.182.110, Interface Address 10.160.0.110
No backup designated router on this network
Timer intervals configured, Hello 10, Dead 40, Wait 40, Retransmit 5
Hello due in 00:00:10
Neighbor Count is 0, Adjacent neighbor count is 0
Crypt Sequence Number is 2081
Hello received 0 sent 41535, DD received 0 sent 0
LS-Req received 0 sent 0, LS-Upd received 0 sent 0
LS-Ack received 0 sent 0, Discarded 0

4.5    If the interface where OSPF should run is not listed above, verify if its subnet is listed in the OSPF prefixes

FGT# get router info protocols

Example output

Routing Protocol is "ospf 0"
Invalid after 0 seconds, hold down 0, flushed after 0
Outgoing update filter list for all interfaces is
Incoming update filter list for all interfaces is
Redistributing:
Routing for Networks:
10.0.0.0/8
Routing Information Sources:
Gateway Distance Last Update
Distance: (default is 110)
Address Mask Distance List

4.6    If OSPF interface state looks good, check HELLO parameters on respective neighbors

Verify the following HELLO parameters on each side : Subnet mask, area number, area type, hello interval, hello dead interval

Note: Both sides must have the same settings to establish an adjacency.

FGT # get router info ospf interface

Example output

Internet Address 10.160.0.110/23, Area 0.0.0.0, MTU 1500
[...]
Timer intervals configured, Hello 10, Dead 40, Wait 40, Retransmit 5
[...]

4.7    Further troubleshooting can be performed by looking at each packet received, sent, and processed, with the the following command .

This command may generate some extensive output; it is also possible to use more specific debug filters instead of "all" to reduce the verbosity.

FGT# diagnose ip router ospf all (enable|disable)
FGT# diagnose ip router ospf level info -> Requested for FortiOS V4.x
FGT# diagnose debug enable

Example output

OSPF: RECV[Hello]: From 192.168.182.106 via external:192.168.182.58 (192.168.182
.106 -> 224.0.0.5)
OSPF: -----------------------------------------------------
OSPF: Header
OSPF: Version 2
OSPF: Type 1 (Hello)
OSPF: Packet Len 52
OSPF: Router ID 192.168.182.106
OSPF: Area ID 0.0.0.0
OSPF: Checksum 0xaa91
OSPF: AuType 0
OSPF: Hello
OSPF: N etworkMask 255.255.254.0
OSPF: HelloInterval 10
OSPF: Options 0x2 (*|-|-|-|-|-|E|-)
OSPF: RtrPriority 1
OSPF: RtrDeadInterval 40
OSPF: DRouter 192.168.182.106
OSPF: BDRouter 192.168.182.58
OSPF: # Neighbors 2
OSPF: Neighbor 192.168.182.58
OSPF: Neighbor 192.168.182.110
OSPF: -----------------------------------------------------
OSPF: NFSM[external:192.168.182.58-192.168.182.106]: Full (HelloReceived)
OSPF: NFSM[external:192.168.182.58-192.168.182.106]: nfsm_ignore called
OSPF: NFSM[external:192.168.182.58-192.168.182.106]: Full (2-WayReceived)
OSPF: LSA[MaxAge]: Maxage walker finished (0.000074 sec)
OSPF: IFSM[internal:10.161.0.58]: Hello timer expire
OSPF: SEND[Hello]: To 224.0.0.5 via internal:10.161.0.58, length 44
OSPF: -----------------------------------------------------
OSPF: Header
OSPF: Version 2
OSPF: Type 1 (Hello)
OSPF: Packet Len 44
OSPF: Router ID 192.168.182.58
OSPF: Area ID 0.0.0.0
OSPF: Checksum 0x7be0
OSPF: AuType 0
OSPF: Hello
OSPF: NetworkMask 255.255.254.0
OSPF: HelloInterval 10
OSPF: Options 0x2 (*|-|-|-|-|-|E|-)
OSPF: RtrPriority 1
OSPF: RtrDeadInterval 40
OSPF: DRouter 10.161.0.58
OSPF: BDRouter 0.0.0.0
OSPF: # Neighbors 0
OSPF: -----------------------------------------------------

5. Recommended procedure to troubleshoot BGP

5.1 Looking for BGP routes in the routing table

FGT# get router info routing-table all

or use a BGP filter:

FGT# get router info routing-table bgp

Example output:

FGT# get router info routing-table all

Codes: K - kernel, C - connected, S - static, R - RIP, B - BGP
O - OSPF, IA - OSPF inter area
N1 - OSPF NSSA external type 1, N2 - OSPF NSSA external type 2
E1 - OSPF external type 1, E2 - OSPF external type 2
i - IS-IS, L1 - IS-IS level-1, L2 - IS-IS level-2, ia - IS-IS inter area
* - candidate default

B* 0.0.0.0/0 [200/0] via 10.142.0.110 (recursive via 10.160.0.74), 2d18h00m
B 1.0.0.0/8 [200/0] via 10.142.0.110 (recursive via 10.160.0.74), 2d18h00m
B 2.0.0.0/8 [200/0] via 10.142.0.110 (recursive via 10.160.0.74), 2d18h00m
O 10.0.0.2/32 [110/110] via 10.160.0.74, dmz, 2d18h20m
C 10.0.0.3/32 is directly connected, loopback

5.2 Check if BGP peering is up

FGT # get router info bgp summary

Example output:

BGP router identifier 192.168.182.106, local AS number 106
BGP table version is 1
0 BGP AS-PATH entries
0 BGP community entries

Neighbor V AS MsgRcvd MsgSent TblVer InQ OutQ Up/Down State/PfxRcd
10.160.0.58 4 58 2 2 0 0 0 00:00:08 0

Total number of neighbors 1

FGT # get router info bgp neighbors

Example output:

BGP neighbor is 10.160.0.58, remote AS 58, local AS 106, external link
BGP version 4, remote router ID 192.168.182.58
BGP state = Established, up for 00:00:17
Last read 00:00:17, hold time is 180, keepalive interval is 60 seconds
Configured hold time is 180, keepalive interval is 60 seconds
Neighbor capabilities:
Route refresh: advertised and received (old and new)
Address family IPv4 Unicast: advertised and received
Received 2 messages, 0 notifications, 0 in queue
Sent 2 messages, 0 notifications, 0 in queue
Route refresh request: received 0, sent 0
Minimum time between advertisement runs is 30 seconds

For address family: IPv4 Unicast
BGP table version 1, neighbor version 0
Index 2, Offset 0, Mask 0x4
Community attribute sent to this neighbor (both)
0 accepted prefixes
0 announced prefixes

Connections established 1; dropped 0
Local host: 10.160.0.106, Local port: 1054
Foreign host: 10.160.0.58, Foreign port: 179
Nexthop: 10.160.0.106

5.3     If no peer(s) can be seen, check L3 connectivity to the peer(s)

FGT # execute ping <peer_ip>

Example output:

FGT # execute ping 10.160.0.58

PING 10.160.0.58 (10.160.0.58): 56 data bytes
64 bytes from 10.160.0.58: icmp_seq=0 ttl=255 time=0.5 ms
64 bytes from 10.160.0.58: icmp_seq=1 ttl=255 time=0.3 ms
64 bytes from 10.160.0.58: icmp_seq=2 ttl=255 time=0.3 ms

5.4    Further troubleshooting can be performed by looking at each packet received, sent, and processed, with the the following command.

This command may generate some extensive output. It is also possible to use more specific debug filters instead of "all" to reduce the verbosity.

FGT # diagnose ip router bgp all (enable|disable)
FGT # diagnose debug enable

Related Articles

Technical Note: Configuring link redundancy - Traffic load-balancing / load-sharing - ECMP (Equal Co...

Technical Tip: FortiGate routing table conditions