Cybersecurity Forum

This forum is for all security enthusiasts to discuss Fortinet's latest & evolving technologies and to connect & network with peers in the cybersecurity hemisphere. Share and learn on a broad range of topics like best practices, use cases, integrations and more. For support specific questions/resources, please visit the Support Forum or the Knowledge Base.

June
New Contributor

Fortigate 5.4.9 Routing Issue

Dear all,
 
I have a question about the Routing Issue.
Routing issues were detected yesterday afternoon in a firewall running on version 5.4.9 of the FortiOS.
Due to current routing issues, there were no service issues, but management access was not possible with each firewall.
 
[firewall Settings]
- FortiGate3100D(5.4 Patch9) - HA : Standalone mode, Vdom : enable
mgmt ip : 192.168.1.6
- FortiGate101E(5.4 Patch9) - HA : Standalone mode, Vdom : enable
mgmt ip : 192.168.1.250
 
[Issue information]
- management access was not possible with each firewall.
- As a result of packet capture at the firewall, the SYN packet which tried GUI Access from the administrator's PC(192.168.120.15) was confirmed. 
However, the firewall does not export SYN + ACK packets.
- We tested ICMP / SSH / HTTPS etc, but the result was the same.
- The target of the "exec traceroute" command of the firewall has been specified as the administrator PC. 
However, the output value was identified as loopback ip with "127.0.0.1".
e.g) exec trace route 
exec trace route 192.168.120.15
traceroute to 192.168.120.15 (192.168.120.15), 32 hops max, 3 probe packets per hot, 84 byte packes
1 127.0.0.1 <localhost> 2991.668 ms !H 3000.442 ms !H^C *

- The applied routing table is as follows.
e.g) config router static
edit 2
set dst 192.168.120.0 255.255.255.0
set gateway 192.168.1.253
set deice mgmt1
next
- And as a result of adding host routing to the routing table, it became normal to communicate.
e.g) config route static
edit 3
set dst 192.168.120.15 255.255.255.255
set gateway 192.168.1.253
set device mgmt1
next
- Also, after deleting the above "edit 3" routing, access was still possible.

- At the time of the problem, "rtcache" had the IP "192.168.1.254" instead of the Default GW IP.
 
### diagnose ip rtcache list
family=02 tab=254 vf=0 type=01 tos=0 flag=00040200
0.0.0.0@0->192.168.120.15@4(mgmt1) gwy=192.168.1.254 prefsrc=192.168.1.6
ci: ref=0 lastused=274 expire=0 err=00000000 used=2 br=0 pmtu=1500
 
### get router info routing-table database
Codes: K - kernel, C - connected, S - static, R - RIP, B - BGP 
O - OSPF, IA - OSPF inter area 
N1 - OSPF NSSA external type 1, N2 - OSPF NSSA external type 2 
E1 - OSPF external type 1, E2 - OSPF external type 2 
i - IS-IS, L1 - IS-IS level-1, L2 - IS-IS level-2, ia - IS-IS inter area 
> - selected route, * - FIB route, p - stale info 

C *> 192.168.1.0/24 is directly connected, mgmt1 
S *> 192.168.120.0/24 [10/0] via 192.168.1.253, mgmt1

### get router info routing-table all
Codes: K - kernel, C - connected, S - static, R - RIP, B - BGP 
O - OSPF, IA - OSPF inter area 
N1 - OSPF NSSA external type 1, N2 - OSPF NSSA external type 2 
E1 - OSPF external type 1, E2 - OSPF external type 2 
i - IS-IS, L1 - IS-IS level-1, L2 - IS-IS level-2, ia - IS-IS inter area 
* - candidate default 

C 192.168.1.0/24 is directly connected, mgmt1 
S 192.168.120.0/24 [10/0] via 192.168.1.253, mgmt1

Is the current symptom a bug?
2 REPLIES 2
YohaDAVI
New Contributor

Hello,

To avoid any confusion and resume :

  * You have two firewall with an interface configured to be used as management service (MGMT if i'm right). Right ?
  * Both firewalls are in same subnet for this management usage. Right ?
  * Issue concerned both firewall and you wasn't able to acces on it from LAN side. Right ?

Moreover, during your troubleshooting, did you do :

  * diagnose firewall iprope flush / execute router restart ==> during issue and before adding a route ?
  * execute ping-options source [ip of your firewall] and then execute ping [gateway ip]
  * diagnose ip arp list ==> during issue to control if firewall had a mac address linked to your gateway. If you firewall wasn't able to resolve mac address linked to your gateway, trafic will not get out interface.
  * diagnose ip route list : do you have this output during issue and after it ?
  * a check of the other routes including policy routes ? Because i don't believe that firewall learn and use 192.168.1.254 as gateway for pleasure... And diagnose ip rtcache list show active sessions already "established", so there is a route linked to this. Just have to clarify "prefsrc=192.168.1.6"...

When you add and when you delete a route, you force firewall to update his routing table. But, in case of add, only new session will be concerned by this new route. I think it's for that reason it was working after add. Moreover, in rtcache, we are able to see that it already exist a route to use 192.168.1.254 instead of 192.168.1.253 so all session will use it. And if firewall wasn't able to resolve mac address linked to 192.168.1.253, your route exist but it's not really "up" in the router.

Currently, i'm not sure that it's a bug. But wait and see...

Another clue : if you have an access on equipment behind your interface (probably a switch), please check logs too on it. Hope you will find something.

------------------------------
Yohann [LastName] [Designation]
Ing?nieur syst?me / r?seaux
[CompanyName]
[City] [State]
[Phone]
------------------------------
June
New Contributor

Hello. Yohann,

* You have two firewall with an interface configured to be used as management service (MGMT if i'm right). Right ?
===> That's right.
* Both firewalls are in same subnet for this management usage. Right ?
===> That's right.
* Issue concerned both firewall and you wasn't able to acces on it from LAN side. Right ?
===> Yes, the GW IP that we don't know remains in the routing cache. It also forwards the response to the request to 127.0.0.1.

* diagnose firewall iprope flush / execute router restart
==> during issue and before adding a route ?
====> The same issue has not occurred so far. You did not perform the command you said when the problem occurred.
* execute ping-options source [ip of your firewall] and then execute ping [gateway ip]
===>No, we tested using the "exec ping" command with no options.

* diagnose ip arp list ==> during issue to control if firewall had a mac address linked to your gateway. If you firewall wasn't able to resolve mac address linked to your gateway, trafic will not get out interface.
===> The following command results were found when the problem occurred:
##diagnose ip arp list
index=4 ifname=mgmt1 192.168.1.253 ac:1f:6b:25:4b:e5 state=00000002 use=1 confirm=1 update=1543 ref=3
index=4 ifname=mgmt1 192.168.1.249 00:21:5a:ec:79:62 state=00000002 use=1 confirm=1654 update
index=4 ifname=mgmt1 192.168.1.254 state=00000020 use=27422 confirm=34022 update=27122 ref=1

* diagnose ip route list : do you have this output during issue and after it ?
===> Sadly, there is no "diagnose ip route list" check result in the tac report file.

* a check of the other routes including policy routes ? Because i don't believe that firewall learn and use 192.168.1.254 as gateway for pleasure... And diagnose ip rtcache list show active sessions already "established", so there is a route linked to this. Just have to clarify "prefsrc=192.168.1.6"...
===> My question is the same. We have no routing overlapped with mgt in the static router configuration.
It also does not use policy routing. But for some reason GW IP was assigned to "rtcahe" as 192.168.1.254?

When you add and when you delete a route, you force firewall to update his routing table. But, in case of add, only new session will be concerned by this new route. I think it's for that reason it was working after add. Moreover, in rtcache, we are able to see that it already exist a route to use 192.168.1.254 instead of 192.168.1.253 so all session will use it. And if firewall wasn't able to resolve mac address linked to 192.168.1.253, your route exist but it's not really "up" in the router.
===> I have the same opinion as you.


As a result of additional questions to the customer,
The GW IP of the firewall using the "192.168.1.253" IP is verified to be an SSL-VPN appliance. (PulseSecure SSL-VPN)
192.168.1.254 is also the GW IP value of the PulseSecure equipment.

However, 192.168.1.254 is not the actual equipment ip address.
Finally, 192.168.1.254 is the Dummy setting(Trash setting) of the internal interface of PulseSecure.

these settings have already been used from the past. But so far no problems have been encountered.
What is the impact of this setting on our equipment?
I do not want to think of it as a problem with our firewall ... but I can not understand the "rtcahe" value.

I need your help and advice.

Thanks and Best Regards,-------------------------------------------
Original Message:
Sent: 12-01-2018 14:42
From: Yohann DAVID
Subject: Fortigate 5.4.9 Routing Issue

Hello,

To avoid any confusion and resume :

  * You have two firewall with an interface configured to be used as management service (MGMT if i'm right). Right ?
  * Both firewalls are in same subnet for this management usage. Right ?
  * Issue concerned both firewall and you wasn't able to acces on it from LAN side. Right ?

Moreover, during your troubleshooting, did you do :

  * diagnose firewall iprope flush / execute router restart ==> during issue and before adding a route ?
  * execute ping-options source [ip of your firewall] and then execute ping [gateway ip]
  * diagnose ip arp list ==> during issue to control if firewall had a mac address linked to your gateway. If you firewall wasn't able to resolve mac address linked to your gateway, trafic will not get out interface.
  * diagnose ip route list : do you have this output during issue and after it ?
  * a check of the other routes including policy routes ? Because i don't believe that firewall learn and use 192.168.1.254 as gateway for pleasure... And diagnose ip rtcache list show active sessions already "established", so there is a route linked to this. Just have to clarify "prefsrc=192.168.1.6"...

When you add and when you delete a route, you force firewall to update his routing table. But, in case of add, only new session will be concerned by this new route. I think it's for that reason it was working after add. Moreover, in rtcache, we are able to see that it already exist a route to use 192.168.1.254 instead of 192.168.1.253 so all session will use it. And if firewall wasn't able to resolve mac address linked to 192.168.1.253, your route exist but it's not really "up" in the router.

Currently, i'm not sure that it's a bug. But wait and see...

Another clue : if you have an access on equipment behind your interface (probably a switch), please check logs too on it. Hope you will find something.

------------------------------
Yohann [LastName] [Designation]
Ing?nieur syst?me / r?seaux
[CompanyName]
[City] [State]
[Phone]
------------------------------

Original Message:
Sent: 12-01-2018 09:53
From: Seongho Jeon
Subject: Fortigate 5.4.9 Routing Issue

Dear all,
 
I have a question about the Routing Issue.
Routing issues were detected yesterday afternoon in a firewall running on version 5.4.9 of the FortiOS.
Due to current routing issues, there were no service issues, but management access was not possible with each firewall.
 
[firewall Settings]
- FortiGate3100D(5.4 Patch9) - HA : Standalone mode, Vdom : enable
mgmt ip : 192.168.1.6
- FortiGate101E(5.4 Patch9) - HA : Standalone mode, Vdom : enable
mgmt ip : 192.168.1.250
 
[Issue information]
- management access was not possible with each firewall.
- As a result of packet capture at the firewall, the SYN packet which tried GUI Access from the administrator's PC(192.168.120.15) was confirmed. 
However, the firewall does not export SYN + ACK packets.
- We tested ICMP / SSH / HTTPS etc, but the result was the same.
- The target of the "exec traceroute" command of the firewall has been specified as the administrator PC. 
However, the output value was identified as loopback ip with "127.0.0.1".
e.g) exec trace route 
exec trace route 192.168.120.15
traceroute to 192.168.120.15 (192.168.120.15), 32 hops max, 3 probe packets per hot, 84 byte packes
1 127.0.0.1 <localhost> 2991.668 ms !H 3000.442 ms !H^C *

- The applied routing table is as follows.
e.g) config router static
edit 2
set dst 192.168.120.0 255.255.255.0
set gateway 192.168.1.253
set deice mgmt1
next
- And as a result of adding host routing to the routing table, it became normal to communicate.
e.g) config route static
edit 3
set dst 192.168.120.15 255.255.255.255
set gateway 192.168.1.253
set device mgmt1
next
- Also, after deleting the above "edit 3" routing, access was still possible.

- At the time of the problem, "rtcache" had the IP "192.168.1.254" instead of the Default GW IP.
 
### diagnose ip rtcache list
family=02 tab=254 vf=0 type=01 tos=0 flag=00040200
0.0.0.0@0->192.168.120.15@4(mgmt1) gwy=192.168.1.254 prefsrc=192.168.1.6
ci: ref=0 lastused=274 expire=0 err=00000000 used=2 br=0 pmtu=1500
 
### get router info routing-table database
Codes: K - kernel, C - connected, S - static, R - RIP, B - BGP 
O - OSPF, IA - OSPF inter area 
N1 - OSPF NSSA external type 1, N2 - OSPF NSSA external type 2 
E1 - OSPF external type 1, E2 - OSPF external type 2 
i - IS-IS, L1 - IS-IS level-1, L2 - IS-IS level-2, ia - IS-IS inter area 
> - selected route, * - FIB route, p - stale info 

C *> 192.168.1.0/24 is directly connected, mgmt1 
S *> 192.168.120.0/24 [10/0] via 192.168.1.253, mgmt1

### get router info routing-table all
Codes: K - kernel, C - connected, S - static, R - RIP, B - BGP 
O - OSPF, IA - OSPF inter area 
N1 - OSPF NSSA external type 1, N2 - OSPF NSSA external type 2 
E1 - OSPF external type 1, E2 - OSPF external type 2 
i - IS-IS, L1 - IS-IS level-1, L2 - IS-IS level-2, ia - IS-IS inter area 
* - candidate default 

C 192.168.1.0/24 is directly connected, mgmt1 
S 192.168.120.0/24 [10/0] via 192.168.1.253, mgmt1

Is the current symptom a bug?