Hello,

Thank you for your question. Based on your description and the fact, that some networks are able to communicate and some not means that there is probably either problem with firewall policy or with some routes withing BGP. First thing what I would do is to run debug flow, to understand what is FortiGate doing:

https://docs.fortinet.com/document/fortigate/6.2.7/cookbook/54688/debugging-the-packet-flow

This debug will tell you, if the traffic is allowed, which should be based on the details provided. Then it will tell you if correct interface is selected as destination. It can also show you if there are any problems with selectors within IPSEC. But my guess is that you have 0.0.0.0/0 as source and destination selectors.

Also the packet capture can tell you if packet is being sent out via correct interface and if reply is received. If you have 2 tunnels, it might be possible that the traffic is going out via VPN1 and reply is coming back via VPN2 and it that case FortiGate might drop the traffic unless some modifications are made. You might test it with only one site to site VPN up and test if the traffic for all destinations will be working.

Thank you for the direction this gave me a bit to work with.

I set up the 80F in a dev environment. It has just wan2 connected and only 1 tunnel to AWS.

Looking at the flow log there was a reference to "Match policy routing id=..." for traffic to the 172.22.0.0/16. This was an SD-WAN Rule telling some networks to favor wan1. I have removed these rules for now but it is something that we want to work out.

id=20085 trace_id=55 func=print_pkt_detail line=5824 msg="vd-root:0 received a packet(proto=1, 192.168.7.2:1->172.22.22.154:2048) tun_id=0.0.0.0 from internal. type=8, code=0, id=1, seq=144."
id=20085 trace_id=55 func=init_ip_session_common line=6003 msg="allocate a new session-00006038, tun_id=0.0.0.0"
id=20085 trace_id=55 func=rpdb_srv_match_input line=1028 msg="Match policy routing id=2133393410: to 172.22.22.154 via ifindex-6"
id=20085 trace_id=55 func=vf_ip_route_input_common line=2604 msg="find a route: flag=04000000 gw-XX.XX.XX.XX via wan2"

Using this map as a summary of the map in the original post:

Device on VPN (172.22.22.154/16)
|
|
OpenVPN Server (10.10.0.1/16 and 172.22.0.1/16)
|
|
Transit Gateway and Site-to-Site VPN
|
|
Meraki (192.168.1.254/24)
|
|
Router/L3 Switch (192.168.1.1/24)
|
|
Workstation (192.168.1.2/24)

Using TCPDump on the OpenVPN Server:

Ping to OpenVPN Server VPC Address

In ethertype IPv4 (0x0800), length 76: 192.168.1.2 > 10.10.0.1: ICMP echo request, id 1, seq 39, length 40
Out ethertype IPv4 (0x0800), length 76: 10.10.0.1 > 192.168.1.2: ICMP echo reply, id 1, seq 39, length 40

Ping to OpenVPN Server VPN Address

In ethertype IPv4 (0x0800), length 76: 192.168.1.2 > 172.22.0.1: ICMP echo request, id 1, seq 43, length 40
Out ethertype IPv4 (0x0800), length 76: 172.22.0.1 > 192.168.1.2: ICMP echo reply, id 1, seq 43, length 40

Ping to Device on VPN

In ethertype IPv4 (0x0800), length 76: 192.168.1.2 > 172.22.22.154: ICMP echo request, id 1, seq 47, length 40
Out ethertype IPv4 (0x0800), length 76: 172.22.0.1 > 172.22.22.154: ICMP echo request, id 1, seq 47, length 40
In ethertype IPv4 (0x0800), length 76: 172.22.22.154 > 172.22.0.1: ICMP echo reply, id 1, seq 47, length 40
Out ethertype IPv4 (0x0800), length 76: 172.22.22.154 > 192.168.1.2: ICMP echo reply, id 1, seq 47, length 40

Basically the above is what we want to see with the new device.

Then testing with the FortiGate in place:

Device on VPN (172.22.22.154/16)
|
|
OpenVPN Server (10.10.0.1/16 and 172.22.0.1/16)
|
|
Transit Gateway and Site-to-Site VPN
|
|
FORTIGATE 80F (192.168.7.254/24) <- Testing network
|
|
Workstation (192.168.7.2) <- Testing network

Using TCPDump on the OpenVPN Server:

Ping to OpenVPN VPC Address

In ethertype IPv4 (0x0800), length 76: 192.168.7.2 > 10.10.0.1: ICMP echo request, id 1, seq 617, length 40
Out ethertype IPv4 (0x0800), length 76: 10.10.0.1 > 192.168.7.2: ICMP echo reply, id 1, seq 617, length 40

When we Ping to OpenVPN VPN Address and Ping to Device on VPN the workstation gets "Request timed out." and TCPDump shows nothing.

These are the results of debug flow on the FortiGate:

On the FortiGate:

diagnose debug flow filter daddr 10.10.0.1
diagnose debug enable
diagnose debug flag trace start

On the Workstation:

tracert 10.10.0.1

Result:

id=20085 trace_id=186 func=print_pkt_detail line=5824 msg="vd-root:0 received a packet(proto=1, 192.168.7.2:1->10.10.0.1:2048) tun_id=0.0.0.0 from internal. type=8, code=0, id=1, seq=639."
id=20085 trace_id=186 func=init_ip_session_common line=6003 msg="allocate a new session-00013f90, tun_id=0.0.0.0"
id=20085 trace_id=186 func=vf_ip_route_input_common line=2604 msg="find a route: flag=04000000 gw-XX.XX.XX.XX via AWS-WAN2-TUN1"
id=20085 trace_id=186 func=fw_forward_handler line=874 msg="Allowed by Policy-12:"

On the FortiGate:

diagnose debug flow filter daddr 172.22.0.1
diagnose debug flag trace start

On the Workstation:

tracert 172.22.0.1

Result:

id=20085 trace_id=185 func=print_pkt_detail line=5824 msg="vd-root:0 received a packet(proto=1, 192.168.7.2:1->172.22.0.1:2048) tun_id=0.0.0.0 from internal. type=8, code=0, id=1, seq=630."
id=20085 trace_id=185 func=init_ip_session_common line=6003 msg="allocate a new session-00013e7a, tun_id=0.0.0.0"
id=20085 trace_id=185 func=vf_ip_route_input_common line=2604 msg="find a route: flag=04000000 gw-XX.XX.XX.XX via AWS-WAN2-TUN1"
id=20085 trace_id=185 func=fw_forward_handler line=874 msg="Allowed by Policy-12:"

• It appears, based on the flow that the traffic is being routed to the tunnel correctly
• The Routes and Security groups are correct because:
  • Existing Meraki traffic works
  • Traffic is making it to the OpenVPN server when using the FortiGate

We reviewed the iptables rules on the OpenVPN Server. These are the rules relevant to our tests:

## 192.168.6.0/23 ##
# ACCEPT ICMP ping to tun0 172.22.0.0/16
iptables -A FORWARD -i eth0 -s 192.168.6.0/23 -p icmp -o tun0 -d 172.22.0.0/16 -j ACCEPT
# ACCEPT already established connections from tun0 172.22.0.0/16
iptables -A FORWARD -m state -i tun0 -s 172.22.0.0/16 -o eth0 -d 192.168.6.0/23 --state ESTABLISHED,RELATED -j ACCEPT
# MASQUERADE Forwards to tun0 172.22.0.0/16 to avoid the need to push a route
iptables -t nat -A POSTROUTING -s 192.168.6.0/23 -o tun0 -d 172.22.0.0/16 -j MASQUERADE

Thank you for the direction this gave me a bit to work with.

I set up the 80F in a dev environment. It has just wan2 connected and only 1 tunnel to AWS.

Looking at the flow log there was a reference to "Match policy routing id=..." for traffic to the 172.22.0.0/16 network. This was an SD-WAN Rule telling some networks to favor wan1. I have removed these rules for now but it is something that we want to work out.

id=20085 trace_id=55 func=print_pkt_detail line=5824 msg="vd-root:0 received a packet(proto=1, 192.168.7.2:1->172.22.22.154:2048) tun_id=0.0.0.0 from internal. type=8, code=0, id=1, seq=144."
id=20085 trace_id=55 func=init_ip_session_common line=6003 msg="allocate a new session-00006038, tun_id=0.0.0.0"
id=20085 trace_id=55 func=rpdb_srv_match_input line=1028 msg="Match policy routing id=2133393410: to 172.22.22.154 via ifindex-6"
id=20085 trace_id=55 func=vf_ip_route_input_common line=2604 msg="find a route: flag=04000000 gw-XX.XX.XX.XX via wan2"

Using this map as a summary of the map in the original post

Device on VPN (172.22.22.154/16)
|
|
OpenVPN Server (10.10.0.1/16 and 172.22.0.1/16)
|
|
Transit Gateway and Site-to-Site VPN
|
|
Meraki (192.168.1.254/24)
|
|
Router/L3 Switch (192.168.1.1/24)
|
|
Workstation (192.168.1.2/24)

Hello,

Based on your last reply, it looks also you have configured SD-WAN in your FGT.
have you looked at the KB below, it might have some useful info for your case

https://community.fortinet.com/t5/FortiGate/Technical-Tip-Configure-FortiGate-SD-WAN-with-an-IPSEC-VPN/ta-p/190756