Cannot ping from lan to lan with ipsec sd-wan

[Tutek]

Hi,

I have configured two ipsec tunnels between Fortigate_A and Fortigate_B with static routing and then added to sd-wan zone on both sides. I created ipv4 allow rules to allow lan to lan traffic, and this is not working, I mean tunnels are up established but cannot ping from lan to lan. So I addressed Ipsec interfaces, now the strange I can ping from Fortigate_B to Fortigate_A ipsec address but not in opposite. Ping are allowed on ipsec interfaces. How to troubleshoot this?

[gfleming]

Sorry I gave wrong command please do output of "diagnose netlink interface list | grep 29"

[Tutek]

FGT # diagnose netlink interface list | grep 29
if=port23 family=00 type=1 index=29 mtu=1500 link=0 master=0
ref=29 state=start present no_carrier fw_flags=8000 flags=up broadcast master multicast 
ref=29 state=start present no_carrier fw_flags=8000 flags=up broadcast multicast 

[Tutek]

 I have downgraded branch fortigate to 6.4.11 and did factory reset, now it is the same version like central fortigate, after configuring it from scratch slowly step by step only with one ipsec tunnel as for now. Now I see that request like www, dns, are routed through the tunnel. I see on sd-wan rules that member have this icon mean that is selected, and hit count increasing on sd-wan-rules:

But on central HQ sd-wan rules that use the same tunnel don't have this icon and hit count not increasing:

Something is wrong with routing on central HQ but don't now how to fix it, everything is configured like in documents :(

[gfleming]

What is on port 24?

And what is output of the following command from central FW?

get router info routing-table details 10.17.2.X/XX 

(where 10.17.2.x/xx is actual remote subnet)

Also it would be very useful if you could not mask your private IPs. There's nothing we can do with them but it helps immensely in troubleshooting so we can identify routing issues.

[Tutek]

FGT # get router info routing-table details 10.17.2.0/24
% Network not in table

but as I said I have summarized remote network in static routes, so:

FGT # get router info routing-table details 10.17.0.0/20

Routing table for VRF=0
Routing entry for 10.17.0.0/20
  Known via "static", distance 10, metric 0, best
  * directly connected, w1-branch-w1 distance 0
  * directly connected, w2-branch-w1 distance 0

[Tutek]

Pings from Central fortigate with source as lan gateway:

FGT # execute ping-options source 10.1.0.1

FGT # execute ping 10.17.2.52
PING 10.17.2.52 (10.17.2.52): 56 data bytes

--- 10.17.2.52 ping statistics ---
5 packets transmitted, 0 packets received, 100% packet loss

branch side:

FGT-branch # diagnose sniffer packet any "host 10.1.0.1" 4
interfaces=[any]
filters=[host 10.1.0.1]
4.687769 w1-centrala-w1 in 10.1.0.1 -> 10.17.2.52: icmp: echo request
5.687762 w1-centrala-w1 in 10.1.0.1 -> 10.17.2.52: icmp: echo request
6.687805 w1-centrala-w1 in 10.1.0.1 -> 10.17.2.52: icmp: echo request
7.687522 w1-centrala-w1 in 10.1.0.1 -> 10.17.2.52: icmp: echo request
8.687361 w1-centrala-w1 in 10.1.0.1 -> 10.17.2.52: icmp: echo request

Now in opposite direction Branch--->Central

FGT-branch # execute ping-options source 10.17.2.1

FGT-rwm2 # execute ping 10.1.0.1
PING 10.1.0.1 (10.1.0.1): 56 data bytes

FGT # diagnose sniffer packet any "host 10.17.2.1" 4
interfaces=[any]
filters=[host 10.17.2.1]
5.676123 w1-branch-w1 in 10.17.2.1 -> 10.1.0.1: icmp: echo request
6.676696 w1-branch-w1 in 10.17.2.1 -> 10.1.0.1: icmp: echo request
7.686798 w1-branch-w1 in 10.17.2.1 -> 10.1.0.1: icmp: echo request
8.695868 w1-branch-w1 in 10.17.2.1 -> 10.1.0.1: icmp: echo request
9.706680 w1-branch-w1 in 10.17.2.1 -> 10.1.0.1: icmp: echo request

[Tutek]

but I can ping some adresses fron branch to central hq lan, take a look:

FGT-branch # execute ping 10.10.0.127
PING 10.10.0.127 (10.10.0.127): 56 data bytes
64 bytes from 10.10.0.127: icmp_seq=0 ttl=127 time=26.8 ms
64 bytes from 10.10.0.127: icmp_seq=1 ttl=127 time=26.8 ms
64 bytes from 10.10.0.127: icmp_seq=2 ttl=127 time=26.7 ms
64 bytes from 10.10.0.127: icmp_seq=3 ttl=127 time=26.8 ms
64 bytes from 10.10.0.127: icmp_seq=4 ttl=127 time=26.8 ms

hq fortigate:

FGT # diagnose sniffer packet any "host 10.10.0.127" 4
interfaces=[any]
filters=[host 10.10.0.127]
0.962899 vlan101 in arp who-has 10.10.0.1 tell 10.10.0.127
8.386607 w1-branch-w1 in 46.x.x.x (wan-ip-branch) -> 10.10.0.127: icmp: echo request
8.386693 vlan101 out 46.x.x.x (wan-ip-branch) -> 10.10.0.127: icmp: echo request
8.386695 lacp out 46.x.x.x (wan-ip-branch) -> 10.10.0.127: icmp: echo request
8.386696 port2 out 46.x.x.x (wan-ip-branch) -> 10.10.0.127: icmp: echo request
8.387151 vlan101 in 10.10.0.127 -> 46.x.x.x (wan-ip-branch): icmp: echo reply
8.387182 w1-branch-w1 out 10.10.0.127 -> 46.x.x.x (wan-ip-branch): icmp: echo reply
9.387182 w1-branch-w1 in 46.x.x.x (wan-ip-branch) -> 10.10.0.127: icmp: echo request
9.387228 vlan101 out 46.x.x.x (wan-ip-branch) -> 10.10.0.127: icmp: echo request
9.387230 lacp out 46.x.x.x (wan-ip-branch) -> 10.10.0.127: icmp: echo request
9.387232 port2 out 46.x.x.x (wan-ip-branch) -> 10.10.0.127: icmp: echo request
9.387670 vlan101 in 10.10.0.127 -> 46.x.x.x (wan-ip-branch): icmp: echo reply
9.387688 w1-branch-w1 out 10.10.0.127 -> 46.x.x.x (wan-ip-branch): icmp: echo reply
10.397287 w1-branch-w1 in 46.x.x.x (wan-ip-branch) -> 10.10.0.127: icmp: echo request
10.397326 vlan101 out 46.x.x.x (wan-ip-branch) -> 10.10.0.127: icmp: echo request
10.397328 lacp out 46.x.x.x (wan-ip-branch) -> 10.10.0.127: icmp: echo request
10.397330 port2 out 46.x.x.x (wan-ip-branch) -> 10.10.0.127: icmp: echo request
10.397872 vlan101 in 10.10.0.127 -> 46.x.x.x (wan-ip-branch): icmp: echo reply
10.397897 w1-branch-w1 out 10.10.0.127 -> 46.x.x.x (wan-ip-branch): icmp: echo reply
11.406986 w1-branch-w1 in 46.x.x.x (wan-ip-branch) -> 10.10.0.127: icmp: echo request
11.407020 vlan101 out 46.x.x.x (wan-ip-branch) -> 10.10.0.127: icmp: echo request

[gfleming]

It's very strange showing source IP of your WAN IP from branch here. It should show the internal IP. And also very strange showing 10.17.X.X network as "directly connected" not as a static route VIA the tunnel interface. Something is very weird here.

[Tutek]

maybe this is because I did not addressed ipsec interfaces, and fortigate pick up the interface with the smallest index?

All my routing table regarding ipsec looking the same, e.g. other ipsec  tunnel that is working:

FGT # get router info routing-table details 10.18.0.0/20

Routing table for VRF=0
Routing entry for 10.18.0.0/20
  Known via "static", distance 10, metric 0, best
  * directly connected, branch2_ipsec distance 0

[Tutek]

I confirmed this, If I set Ip on ipsec tunnels, then when sniff incoming request there is no more remote branch wan IP, is instead tunnel interface IP (1.1.1.2):

FGT # diagnose sniffer packet any "host 10.10.0.102 and icmp" 4 0 a
interfaces=[any]
filters=[host 10.10.0.102 and icmp]
2022-11-27 18:33:21.032778 w1-branch-w1 in 1.1.1.2 -> 10.10.0.102: icmp: echo request
2022-11-27 18:33:28.809738 w1-branch-w1 in 1.1.1.2 -> 10.10.0.102: icmp: echo request
2022-11-27 18:33:29.813914 w1-branch-w1 in 1.1.1.2 -> 10.10.0.102: icmp: echo request