Site to Site VPN odd behaviour

[KCSRog]

Hi everyone, new to the forums so be gentle...

I've got an odd problem with a site to site IPSEC VPN between two Fortigate devices.

FGT60E - Site 1 IP: 192.168.3.240 Local Subnet: 192.168.3.0/24

FGT30E - Site 2 IP: 192.168.4.254 Local Subnet: 192.168.4.0/24

The tunnel is up and I can communicate between the two sites partially. Some ips are visible and others aren't which has got me baffled. I've tried a bit of troubleshooting but I'm fairly new to Fortigate so bear with me.

I've got static routes in place to pass traffic down the VPN tunnel and firewall rules to allow traffic in and out at both sites.

I've grabbed some test commands from a few places but I'm not sure what it's telling me.

Running a test from site 2 to site 1, originating from 192.168.4.135 and heading to 192.168.4.241.

On Site 1 cli I ran the following:

diag sniffer packet any 'host 192.168.3.241 and icmp' 6 0 a

interfaces=[any] filters=[host 192.168.3.241 and icmp] 2019-12-19 16:10:17.058440 VPN in 192.168.4.135 -> 192.168.3.241: icmp: echo request 0x0000 0000 0000 0001 0000 0000 0000 0800 4500 ..............E. 0x0010 003c d71e 0000 7f01 dad9 c0a8 0487 c0a8 .<.............. 0x0020 03f1 0800 46af 0001 06ac 6162 6364 6566 ....F.....abcdef 0x0030 6768 696a 6b6c 6d6e 6f70 7172 7374 7576 ghijklmnopqrstuv 0x0040 7761 6263 6465 6667 6869 wabcdefghi 2019-12-19 16:10:17.058852 internal out 192.168.4.135 -> 192.168.3.241: icmp: echo request 0x0000 0000 0000 0000 04d5 9052 5421 0800 4500 .........RT!..E. 0x0010 003c d71e 0000 7e01 dbd9 c0a8 0487 c0a8 .<....~......... 0x0020 03f1 0800 46af 0001 06ac 6162 6364 6566 ....F.....abcdef 0x0030 6768 696a 6b6c 6d6e 6f70 7172 7374 7576 ghijklmnopqrstuv 0x0040 7761 6263 6465 6667 6869 wabcdefghi 2019-12-19 16:10:21.930629 VPN in 192.168.4.135 -> 192.168.3.241: icmp: echo request 0x0000 0000 0000 0001 0000 0000 0000 0800 4500 ..............E. 0x0010 003c d71f 0000 7f01 dad8 c0a8 0487 c0a8 .<.............. 0x0020 03f1 0800 46ae 0001 06ad 6162 6364 6566 ....F.....abcdef 0x0030 6768 696a 6b6c 6d6e 6f70 7172 7374 7576 ghijklmnopqrstuv 0x0040 7761 6263 6465 6667 6869 wabcdefghi 2019-12-19 16:10:21.930681 internal out 192.168.4.135 -> 192.168.3.241: icmp: echo request 0x0000 0000 0000 0000 04d5 9052 5421 0800 4500 .........RT!..E. 0x0010 003c d71f 0000 7e01 dbd8 c0a8 0487 c0a8 .<....~......... 0x0020 03f1 0800 46ae 0001 06ad 6162 6364 6566 ....F.....abcdef 0x0030 6768 696a 6b6c 6d6e 6f70 7172 7374 7576 ghijklmnopqrstuv 0x0040 7761 6263 6465 6667 6869 wabcdefghi

I ran the following from the cli on Site 1. 

diag debug reset diag debug disable diag debug flow filter addr 192.168.3.241 diag debug flow filter proto 1 diag debug flow trace start 100 diag debug enable

id=20085 trace_id=2 func=print_pkt_detail line=5460 msg="vd-root:0 received a packet(proto=1, 192.168.4.135:1->192.168.3.241:2048) from VPN. type=8, code=0, id=1, seq=901." id=20085 trace_id=2 func=resolve_ip_tuple_fast line=5540 msg="Find an existing session, id-0006ad5e, original direction" id=20085 trace_id=3 func=print_pkt_detail line=5460 msg="vd-root:0 received a packet(proto=1, 192.168.4.135:1->192.168.3.241:2048) from VPN. type=8, code=0, id=1, seq=902." id=20085 trace_id=3 func=resolve_ip_tuple_fast line=5540 msg="Find an existing session, id-0006ad5e, original direction" id=20085 trace_id=4 func=print_pkt_detail line=5460 msg="vd-root:0 received a packet(proto=1, 192.168.4.135:1->192.168.3.241:2048) from VPN. type=8, code=0, id=1, seq=903." id=20085 trace_id=4 func=resolve_ip_tuple_fast line=5540 msg="Find an existing session, id-0006ad5e, original direction" id=20085 trace_id=5 func=print_pkt_detail line=5460 msg="vd-root:0 received a packet(proto=1, 192.168.4.135:1->192.168.3.241:2048) from VPN. type=8, code=0, id=1, seq=904." id=20085 trace_id=5 func=resolve_ip_tuple_fast line=5540 msg="Find an existing session, id-0006ad5e, original direction"

Can anyone give me some guidance on where to look?

[Toshi_Esumi]

It's just showing the Site1 FGT is receiving the ping requests over VPN and forwarding them to internal interface normally. But it's not getting any replies back from the destination.

I would check at the destination device if it's receiving and/or sending replies with Wireshark or something.

[ede_pfau]

Look at the traffic on the LAN port of site1, with the sniffer. BTW, level 4 is sufficient for sniffing ICMP, there's not much in the payload.

I suspect the receiving host doesn't know how to handle the 'foreign' source address of that traffic. You could enable NAT on the site1-policy from VPN to LAN to see if replies come on then. This would indicate there is a routing problem on the host.

For the erratic behavior, check if you can attribute go/nogo to even/odd source or destination addresses, or if there is no obvious correlation. Generally, debug flow and sniffer should clearly show you what is going on.