FortiGate: a specific WAN interface fail with "execute ping"

[SonHuynh]

Hi,

I have 3 WAN interfaces:

WAN1: PPPOE

WAN2: Trunk port (with 2 subInterfaces + Public IPs)

WAN3: PPPOE

When all of the WANs are functioning properly, I use the CLI on the FortiGate:

"execute ping-options source <IP of WAN interface>"

and try

"execute ping 8.8.8.8"

Then, only WAN1 and WAN3 are successful. WAN2 fails.

I also tried pinging from a PC inside the internal network, and it applied an SD-WAN rule to route all traffic through WAN2, which was successful.

I don't know what is going on.

Could anyone help with troubleshooting the case?"

[srajeswaran]

Can you open two CLI sessions and execute ping from 1 and do a sniffer on the other terminal. This will confirm if the ICMP packets are leaving Fortigate and whats is the source address it is taking and then if there is a reply coming back.

diagnose sniffer packet any "host 8.8.8.8" 4

[hbac]

Hi @SonHuynh,

Please make sure you set the correct source IP? Are you able to ping the default gateway of wan2? You can configure link monitor for wan2 and see if it's successful. Please refer to https://community.fortinet.com/t5/FortiGate/Technical-Tip-Link-monitor/ta-p/197504

config system link-monitor
edit "1"
set srcintf "wan2"
set server "8.8.8.8" 
next
end

Regards, 

[SonHuynh]

I've tried some scenarios, and there's a new process.

When i remove out WAN1, and WAN3 from SD-WAN, only WAN2 ( 2 sub interfaces). Then the ping-option with WAN2 IPs is a success and i don't know why!

I also can not clearly understand the results.

Could anyone give me a suggestion?

ps: the WAN IP had changed to keep it secret.

FTG-08 # diagnose sniffer packet any "host 1.1.1.1" 4
interfaces=[any]
filters=[host 1.1.1.1]

//Ping from internal PC (192.168.51.62) to 1.1.1.1
3.993525 GateWay VLAN51 in 192.168.51.62 -> 1.1.1.1: icmp: echo request
3.993697 wan2.sub-interface-1 out 11.11.11.250 -> 1.1.1.1: icmp: echo request
3.993707 wan2 out 11.11.11.250 -> 1.1.1.1: icmp: echo request
4.020729 wan2.sub-interface-1 in 1.1.1.1 -> 11.11.11.250: icmp: echo reply
4.020773 GateWay VLAN51 out 1.1.1.1 -> 192.168.51.62: icmp: echo reply
4.020781 internal out 1.1.1.1 -> 192.168.51.62: icmp: echo reply
5.005622 GateWay VLAN51 in 192.168.51.62 -> 1.1.1.1: icmp: echo request
5.005684 wan2.sub-interface-1 out 11.11.11.250 -> 1.1.1.1: icmp: echo request
5.005691 wan2 out 11.11.11.250 -> 1.1.1.1: icmp: echo request
5.032747 wan2.sub-interface-1 in 1.1.1.1 -> 11.11.11.250: icmp: echo reply
5.032789 GateWay VLAN51 out 1.1.1.1 -> 192.168.51.62: icmp: echo reply
5.032797 internal out 1.1.1.1 -> 192.168.51.62: icmp: echo reply


//Execute ping-option source 11.11.11.242, in FortiGate ClI
//Execute ping 1.1.1.1
52.147889 ppp2 out 11.11.11.242 -> 1.1.1.1: icmp: echo request
53.154638 ppp2 out 11.11.11.242 -> 1.1.1.1: icmp: echo request
54.164549 ppp2 out 11.11.11.242 -> 1.1.1.1: icmp: echo request
55.174557 ppp2 out 11.11.11.242 -> 1.1.1.1: icmp: echo request
56.184558 ppp2 out 11.11.11.242 -> 1.1.1.1: icmp: echo request


//Execute ping-option source 11.11.11.242, in FortiGate ClI when remote WAN1, WAN3 from SDWAN
//Execute ping 1.1.1.1
598.125342 wan2.sub-interface-1 out 11.11.11.242 -> 1.1.1.1: icmp: echo request
598.125359 wan2 out 11.11.11.242 -> 1.1.1.1: icmp: echo request
598.152315 wan2.sub-interface-2 in 1.1.1.1 -> 11.11.11.242: icmp: echo reply
599.134688 wan2.sub-interface-1 out 11.11.11.242 -> 1.1.1.1: icmp: echo request
599.134697 wan2 out 11.11.11.242 -> 1.1.1.1: icmp: echo request
599.161632 wan2.sub-interface-2 in 1.1.1.1 -> 11.11.11.242: icmp: echo reply
600.144666 wan2.sub-interface-1 out 11.11.11.242 -> 1.1.1.1: icmp: echo request
600.144675 wan2 out 11.11.11.242 -> 1.1.1.1: icmp: echo request
600.171704 wan2.sub-interface-2 in 1.1.1.1 -> 11.11.11.242: icmp: echo reply
601.154885 wan2.sub-interface-1 out 11.11.11.242 -> 1.1.1.1: icmp: echo request
601.154899 wan2 out 11.11.11.242 -> 1.1.1.1: icmp: echo request
601.181888 wan2.sub-interface-2 in 1.1.1.1 -> 11.11.11.242: icmp: echo reply
602.164661 wan2.sub-interface-1 out 11.11.11.242 -> 1.1.1.1: icmp: echo request
602.164670 wan2 out 11.11.11.242 -> 1.1.1.1: icmp: echo request
602.191612 wan2.sub-interface-2 in 1.1.1.1 -> 11.11.11.242: icmp: echo reply


//Execute ping-option source 11.11.11.250, in FortiGate ClI when remote WAN1, WAN3 from SDWAN
//Execute ping 1.1.1.1
1400.558041 softnet-3105 out 11.11.11.250 -> 1.1.1.1: icmp: echo request
1400.558060 wan2 out 11.11.11.250 -> 1.1.1.1: icmp: echo request
1400.580912 softnet-3104 in 1.1.1.1 -> 11.11.11.250: icmp: echo reply
1401.564851 softnet-3105 out 11.11.11.250 -> 1.1.1.1: icmp: echo request
1401.564859 wan2 out 11.11.11.250 -> 1.1.1.1: icmp: echo request
1401.587822 softnet-3104 in 1.1.1.1 -> 11.11.11.250: icmp: echo reply
1402.574853 softnet-3105 out 11.11.11.250 -> 1.1.1.1: icmp: echo request
1402.574861 wan2 out 11.11.11.250 -> 1.1.1.1: icmp: echo request
1402.604154 softnet-3104 in 1.1.1.1 -> 11.11.11.250: icmp: echo reply
1403.584858 softnet-3105 out 11.11.11.250 -> 1.1.1.1: icmp: echo request
1403.584868 wan2 out 11.11.11.250 -> 1.1.1.1: icmp: echo request
1403.607833 softnet-3104 in 1.1.1.1 -> 11.11.11.250: icmp: echo reply
1404.594834 softnet-3105 out 11.11.11.250 -> 1.1.1.1: icmp: echo request
1404.594842 wan2 out 11.11.11.250 -> 1.1.1.1: icmp: echo request
1404.618015 softnet-3104 in 1.1.1.1 -> 11.11.11.250: icmp: echo reply

[DPadula]

Hi SonHuynh,

Are the WAN1, 2 and 3 part of the same SD-WAN interface? 
If you can share the settings on SD-WAN and also the routing table it would be helpful. 

Ping has a special option to use the sd-wan interface. You can try and let us know the outcome. 

execute ping-options use-sdwan yes

chewbacca-kvm54 # execute ping-options view-settings
Ping Options:
Repeat Count: 5
Data Size: 56
Timeout: 2
Interface: auto
Interval: 1
TTL: 64
TOS: 0
DF bit: unset
Source Address: auto
VRF: 0
Pattern:
Pattern Size in Bytes: 0
Validate Reply: no
Adaptive Ping: disable
Use SD-WAN: yes

[SonHuynh]

Hi @DPadula ,

I tried your guide and the outcome below:

//SDWAN config
config system sdwan
    set status enable
    config zone
        edit "virtual-wan-link"
        next
        edit "SDWAN-01"
        next
    end
    config members
        edit 2
            set interface "internal7"
        next
        edit 3
            set interface "wan1"
            set zone "SDWAN-01"
        next
        edit 5
            set interface "wan3"
            set zone "SDWAN-01"
        next
        edit 7
            set interface "wan2.sub-interface-2"
            set zone "SDWAN-01"
            set gateway 11.11.11.249
        next
        edit 8
            set interface "wan2.sub-interface-1"
            set zone "SDWAN-01"
            set gateway 11.11.11.241
        next
    end
	
//Routing table
FTG08 # get router info routing-table all
Codes: K - kernel, C - connected, S - static, R - RIP, B - BGP
       O - OSPF, IA - OSPF inter area
       N1 - OSPF NSSA external type 1, N2 - OSPF NSSA external type 2
       E1 - OSPF external type 1, E2 - OSPF external type 2
       i - IS-IS, L1 - IS-IS level-1, L2 - IS-IS level-2, ia - IS-IS inter area
       * - candidate default

Routing table for VRF=0
S*      0.0.0.0/0 [1/0] via 11.11.11.249, wan2.sub-interface-2, [1/0]
                  [1/0] via 11.11.11.241, wan2.sub-interface-1, [1/0]
                  [1/0] via 23.29.8.28, ppp2, [1/0]
                  [1/0] via 23.29.8.28, ppp3, [1/0]
C       10.10.10.0/24 is directly connected, VLAN53
S       10.86.2.0/24 [10/0] via CompanyNameSZ tunnel 122.192.51.106, [1/0]
S       10.86.5.0/24 [10/0] via CompanyNameSZ tunnel 122.192.51.106, [1/0]
S       10.86.9.0/24 [10/0] via CompanyNameSZ tunnel 122.192.51.106, [1/0]
S       10.86.18.0/24 [10/0] via CompanyNameSZ tunnel 122.192.51.106, [1/0]
C       10.100.11.0/24 is directly connected, VLAN52
C       104.176.231.86/32 is directly connected, ppp3
C       104.176.232.2/32 is directly connected, ppp2
C       11.11.11.240/29 is directly connected, wan2.sub-interface-1
C       11.11.11.248/29 is directly connected, wan2.sub-interface-2
S       122.192.51.106/32 [10/0] via 23.29.8.28, wan1, [1/0]
C       23.29.8.28/32 is directly connected, ppp2
                       is directly connected, ppp3
C       169.254.1.1/32 is directly connected, IPSEC VPN
C       192.168.50.0/24 is directly connected, GateWay VLAN 50
C       192.168.51.0/24 is directly connected, GateWay VLAN51




*********************8
// Enable ping use-sdwan

FTG08 # execute ping-options use-sdwan yes

FTG08 # execute ping 1.1.1.1
PING 1.1.1.1 (1.1.1.1): 56 data bytes
64 bytes from 1.1.1.1: icmp_seq=0 ttl=60 time=19.5 ms
64 bytes from 1.1.1.1: icmp_seq=1 ttl=60 time=19.7 ms
64 bytes from 1.1.1.1: icmp_seq=2 ttl=60 time=19.2 ms
64 bytes from 1.1.1.1: icmp_seq=3 ttl=60 time=19.5 ms
64 bytes from 1.1.1.1: icmp_seq=4 ttl=60 time=19.2 ms

--- 1.1.1.1 ping statistics ---
5 packets transmitted, 5 packets received, 0% packet loss
round-trip min/avg/max = 19.2/19.4/19.7 ms

FTG08 # execute ping-options source 11.11.11.250

FTG08 # execute ping 1.1.1.1
PING 1.1.1.1 (1.1.1.1): 56 data bytes

--- 1.1.1.1 ping statistics ---
5 packets transmitted, 0 packets received, 100% packet loss

FTG08 # execute ping-options view-settings 
Ping Options:
        Repeat Count: 5
        Data Size: 56
        Timeout: 2
        Interface: auto
        Interval: 1
        TTL: 64
        TOS: 0
        DF bit: unset
        Source Address: 11.11.11.250
        VRF: 0
        Pattern: 
        Pattern Size in Bytes: 0
        Validate Reply: no
        Adaptive Ping: disable
        Use SD-WAN: yes
		
		
//Sniffer packet of ping with sdwan enable
FTG08 # diagnose sniffer packet any "host 1.1.1.1" 4
interfaces=[any]
filters=[host 1.1.1.1]
1271.496636 ppp2 out 104.176.232.2 -> 1.1.1.1: icmp: echo request
1271.515994 ppp2 in 1.1.1.1 -> 104.176.232.2: icmp: echo reply
1272.511224 ppp2 out 104.176.232.2 -> 1.1.1.1: icmp: echo request
1272.530916 ppp2 in 1.1.1.1 -> 104.176.232.2: icmp: echo reply
1273.521276 ppp2 out 104.176.232.2 -> 1.1.1.1: icmp: echo request
1273.540452 ppp2 in 1.1.1.1 -> 104.176.232.2: icmp: echo reply
1274.531252 ppp2 out 104.176.232.2 -> 1.1.1.1: icmp: echo request
1274.550728 ppp2 in 1.1.1.1 -> 104.176.232.2: icmp: echo reply
1275.541246 ppp2 out 104.176.232.2 -> 1.1.1.1: icmp: echo request
1275.560441 ppp2 in 1.1.1.1 -> 104.176.232.2: icmp: echo reply
1303.762746 ppp3 out 11.11.11.250 -> 1.1.1.1: icmp: echo request
1304.771286 ppp3 out 11.11.11.250 -> 1.1.1.1: icmp: echo request
1305.781301 ppp3 out 11.11.11.250 -> 1.1.1.1: icmp: echo request
1306.791322 ppp3 out 11.11.11.250 -> 1.1.1.1: icmp: echo request
1307.801320 ppp3 out 11.11.11.250 -> 1.1.1.1: icmp: echo request

[hbac]

@SonHuynh,

Have you tried to ping the default gateway of wan2? Why do you have two wan2.sub-interface in the same subnet? In the packet capture, it is showing outgoing traffic wan2.sub-interface-1 but incoming traffic on wan2.sub-interface-2. The same happened on softnet-3104 and softnet-3105. 

Regards,

[SonHuynh]

Yes, I did. It's successful when ping to the gateways in WAN2.

There are 2 subnets:

Sub-Interface name	    Network 	   Usable Host Range	      Gateway
wan2.sub-interface-1	11.11.11.240/29	11.11.11.241 -246	11.11.11.241
wan2.sub-interface-2	11.11.11.248/29	11.11.11.249 - 254	11.11.11.249

I don't know why the traffic in/out way is mismacth when i set ping-option source.

//ping 1.1.1.1 from source 11.11.11.250 ( wan2.sub-interface-2)
FTG08 # diagnose sniffer packet any "host 1.1.1.1" 4
interfaces=[any]
filters=[host 1.1.1.1]
8.664397 wan2.sub-interface-2 out 11.11.11.250 -> 1.1.1.1: icmp: echo request
8.664414 wan2 out 11.11.11.250 -> 1.1.1.1: icmp: echo request
8.687406 wan2.sub-interface-1 in 1.1.1.1 -> 11.11.11.250: icmp: echo reply
9.676723 wan2.sub-interface-2 out 11.11.11.250 -> 1.1.1.1: icmp: echo request
9.676738 wan2 out 11.11.11.250 -> 1.1.1.1: icmp: echo request
9.699811 wan2.sub-interface-1 in 1.1.1.1 -> 11.11.11.250: icmp: echo reply
10.686723 wan2.sub-interface-2 out 11.11.11.250 -> 1.1.1.1: icmp: echo request
10.686741 wan2 out 11.11.11.250 -> 1.1.1.1: icmp: echo request
10.709800 wan2.sub-interface-1 in 1.1.1.1 -> 11.11.11.250: icmp: echo reply
11.696688 wan2.sub-interface-2 out 11.11.11.250 -> 1.1.1.1: icmp: echo request
11.696703 wan2 out 11.11.11.250 -> 1.1.1.1: icmp: echo request
11.719660 wan2.sub-interface-1 in 1.1.1.1 -> 11.11.11.250: icmp: echo reply
12.706716 wan2.sub-interface-2 out 11.11.11.250 -> 1.1.1.1: icmp: echo request
12.706734 wan2 out 11.11.11.250 -> 1.1.1.1: icmp: echo request
12.729685 wan2.sub-interface-1 in 1.1.1.1 -> 11.11.11.250: icmp: echo reply

//ping 1.1.1.1 from source 11.11.11.242 ( wan2.sub-interface-1)
146.281898 wan2.sub-interface-1 out 11.11.11.242 -> 1.1.1.1: icmp: echo request
146.281913 wan2 out 11.11.11.242 -> 1.1.1.1: icmp: echo request
146.304867 wan2.sub-interface-2 in 1.1.1.1 -> 11.11.11.242: icmp: echo reply
147.286748 wan2.sub-interface-1 out 11.11.11.242 -> 1.1.1.1: icmp: echo request
147.286758 wan2 out 11.11.11.242 -> 1.1.1.1: icmp: echo request
147.309675 wan2.sub-interface-2 in 1.1.1.1 -> 11.11.11.242: icmp: echo reply
148.296731 wan2.sub-interface-1 out 11.11.11.242 -> 1.1.1.1: icmp: echo request
148.296740 wan2 out 11.11.11.242 -> 1.1.1.1: icmp: echo request
148.319663 wan2.sub-interface-2 in 1.1.1.1 -> 11.11.11.242: icmp: echo reply
149.306763 wan2.sub-interface-1 out 11.11.11.242 -> 1.1.1.1: icmp: echo request
149.306775 wan2 out 11.11.11.242 -> 1.1.1.1: icmp: echo request
149.329696 wan2.sub-interface-2 in 1.1.1.1 -> 11.11.11.242: icmp: echo reply
150.316733 wan2.sub-interface-1 out 11.11.11.242 -> 1.1.1.1: icmp: echo request
150.316741 wan2 out 11.11.11.242 -> 1.1.1.1: icmp: echo request
150.339661 wan2.sub-interface-2 in 1.1.1.1 -> 11.11.11.242: icmp: echo reply

[DPadula]

Hi SonHuynh,

SD-WAN settings and routing table look correct. Definitely there is something wrong with WAN2 that we couldn't see with the tests done and settings provided. 

I suggest you to raise a support ticket then the TAC team will schedule a remote session for further troubleshoot. 
I have 2 more questions that you might have answered already. 

If I recall properly, you said that with WAN2 out of the SDWAN the connectivity works fine, is that correct?
Assuming that it is not correct, have you tried to connect a computer to the WAN2 link to confirm if you can access Internet from it? This test will us if the issue is on Fortigate or on WAN2 link. 




 

[SonHuynh]

Hi @DPadula,

When i remove out WAN1 and WAN3 of out SDWAN, only WAN2 in SDWAN then ping-option source IP_WAN2 is a success.

Yes, when I applied SDWAN rule to a PC connected to Internet through WAN2 and it was still fine.