Trouble routing to a network beyond AWS Site-to-Site VPN and Transit Gateway

[night_admin]

Background

We are in the process of replacing a Meraki device with a FortiGate 80F. We attempted to migrate over but had to roll back after a critical service was not accessible.

Current Setup

We have IOT devices that use OpenVPN to connect to an OpenVPN server in AWS. Our AWS VPCs are attached to a Transit Gateway. We use AWS Site-to-Site VPN with Static routing to connect our Office to the Transit Gateway.

Hopefully this make sense: (There are other Subnets and VPCs so I have only included the relevant ones)

An IOT device on OpenVPN
172.22.22.154/16
|
|
OpenVPN Server
172.22.0.1/16 (VPN Address) - 10.10.0.1/16 (VPC Address)
|
|
VPC Route Table
172.22.0.0/16 -> (eni of OpenVPN Server)
10.10.0.0/16 -> Local VPC
192.168.1.0/24 -> Transit Gateway
192.168.2.0/24 -> Transit Gateway
|
|
Transit Gateway Route Table
10.10.0.0/16 -> VPC1 Attachment (Propagated)
172.22.0.0/16 -> VPC1 Attachment (Static)
192.168.1.0/24 -> VPN Attachment (Static)
192.168.2.0/24 -> VPN Attachment (Static)
|
|
AWS Transit Gateway
|
|
AWS Site to Site VPN (only one tunnel) Static Routes
|
|
AWS Customer Gateway
|
|
Meraki Device
WAN: 110.x.x.1/30 -> (it's gateway is 110.x.x.2)
LAN: 192.168.1.254/24
Route Table:
10.10.0.0/16 -> Site-to-Site VPN (Static)
172.22.0.0/16 -> Site-to-Site VPN (Static)
|                   |
|                   |
Router (L3 Switch)  Workstation on Remote Worker VPN (l2tp over IPsec)
192.168.1.1         192.168.2.1/24
|
|
Workstation
192.168.1.2

• Computers on the 192.168.1.0/24 or 192.168.2.0/24 networks are able to communicate with devices on the IOT VPN 172.22.0.0/16
• Computers on the 192.168.1.0/24 or 192.168.2.0/24 networks are able to communicate with the OpenVPN Server 10.10.0.1 in the AWS VPC

FortiGate Setup

Over the weekend we swapped the Meraki Device with the FortiGate 80F. We created a new Site-to-Site VPN in AWS using Dynamic routing this time. The Client Remote Access VPN that was on the Meraki was replicated on the FortiGate. All was going well until trying to communicate with a device on the 172.22.0.0/24 network.

This is the new setup: (Differences are in Bold)

An IOT device on OpenVPN
172.22.22.154/16
|
|
OpenVPN Server
172.22.0.1/16 - 10.10.0.1/16
|
|
VPC Route Table
172.22.0.0/16 -> (eni of OpenVPN Server)
10.10.0.0/16 -> Local VPC
192.168.1.0/24 -> Transit Gateway
192.168.2.0/24 -> Transit Gateway
|
|
Transit Gateway Route Table
10.10.0.0/16 -> VPC1 Attachment (Propagated)
172.22.0.0/16 -> VPC1 Attachment (Static)
192.168.1.0/24 -> VPN Attachment (Propagated)
192.168.2.0/24 -> VPN Attachment (Propagated)
|
|
AWS Transit Gateway
|
|
AWS Site to Site VPN (both tunnels) Dynamic Routing
|
|
AWS Customer Gateway
|
|
FortiGate 80F 7.0.5
WAN: 110.x.x.1/30 -> (it's gateway is 110.x.x.2)
LAN: 192.168.1.254/24
Route Table:
10.10.0.0/16 -> Site-to-Site VPN (BGP)
172.22.0.0/16 -> Site-to-Site VPN (BGP)
The 2 tunnels are in an SD-WAN Zone together
|                   |
|                   |
Router (L3 Switch)  Workstation on Remote Worker VPN (l2tp over IPsec)
192.168.1.1         192.168.2.1/24
|
|
Workstation
192.168.1.2

• Computers on the 192.168.1.0/24 network ARE NOT able to communicate with devices on the IOT VPN 172.22.0.0/16
• Computers on the 192.168.1.0/24 network ARE able to communicate with the OpenVPN Server 10.10.0.1 in the AWS VPC
• Computers on the 192.168.2.0/24 (Client VPN to FortiGate) network ARE able communicate with devices on the IOT VPN 172.22.0.0/16
• Computers on the 192.168.2.0/24 (Client VPN to FortiGate) network ARE able to communicate with the OpenVPN Server 10.10.0.1 in the AWS VPC

I don't have it in place at the moment so cant get full details but this what I saw from the 192.168.1.0/24 network.

diag sniffer packet any 'host 10.10.0.1 and icmp' 4 0

-> AWS VPN Tunnel

diag sniffer packet any 'host 172.22.22.154 and icmp' 4 0

-> wan1

When I do a route lookup on the FortiGate to 172.22.22.154 it shows the BGP route to 172.22.0.0/16 via Tunnel 1 and Tunnel 2

The firewall has:

From         | To           | Source         | Destination    | NAT
Internal     | AWS VPN Zone | 192.168.1.0/24 | 10.10.0.0/16   | No
                                               172.22.0.0/16
AWS VPN Zone | Internal     | 10.10.0.0/16   | 192.168.1.0/24 | No
                              172.22.0.0/16
l2t.root     | AWS VPN Zone | 192.168.2.0/24 | 10.10.0.0/16   | No
                                               172.22.0.0/16
AWS VPN Zone | l2t.root     | 10.10.0.0/16   | 192.168.2.0/24 | No
                              172.22.0.0/16

We are new to Fortinet so would some guidance on where to go from here would be amazing.

[akristof]

Hello,

Thank you for your question. Based on your description and the fact, that some networks are able to communicate and some not means that there is probably either problem with firewall policy or with some routes withing BGP. First thing what I would do is to run debug flow, to understand what is FortiGate doing:

https://docs.fortinet.com/document/fortigate/6.2.7/cookbook/54688/debugging-the-packet-flow

This debug will tell you, if the traffic is allowed, which should be based on the details provided. Then it will tell you if correct interface is selected as destination. It can also show you if there are any problems with selectors within IPSEC. But my guess is that you have 0.0.0.0/0 as source and destination selectors.

Also the packet capture can tell you if packet is being sent out via correct interface and if reply is received. If you have 2 tunnels, it might be possible that the traffic is going out via VPN1 and reply is coming back via VPN2 and it that case FortiGate might drop the traffic unless some modifications are made. You might test it with only one site to site VPN up and test if the traffic for all destinations will be working.

[night_admin]

Thank you for the direction this gave me a bit to work with.

I set up the 80F in a dev environment. It has just wan2 connected and only 1 tunnel to AWS.

Looking at the flow log there was a reference to "Match policy routing id=..." for traffic to the 172.22.0.0/16 network. This was an SD-WAN Rule telling some networks to favor wan1. I have removed these rules for now but it is something that we want to work out.

id=20085 trace_id=55 func=print_pkt_detail line=5824 msg="vd-root:0 received a packet(proto=1, 192.168.7.188:1->172.22.22.129:2048) tun_id=0.0.0.0 from internal. type=8, code=0, id=1, seq=144."
id=20085 trace_id=55 func=init_ip_session_common line=6003 msg="allocate a new session-00006038, tun_id=0.0.0.0"
id=20085 trace_id=55 func=rpdb_srv_match_input line=1028 msg="Match policy routing id=2133393410: to 172.22.22.129 via ifindex-6"
id=20085 trace_id=55 func=vf_ip_route_input_common line=2604 msg="find a route: flag=04000000 gw-10.0.0.1 via wan2"

Using this map as a summary of the map in the original post:

Device on VPN (172.22.22.154/16)
|
|
OpenVPN Server (10.10.0.1/16 and 172.22.0.1/16)
|
|
Transit Gateway and Site-to-Site VPN
|
|
Meraki (192.168.1.254/24)
|
|
Router/L3 Switch (192.168.1.1/24)
|
|
Workstation (192.168.1.2)

We used TCPDump on the OpenVPN Server:

Ping to OpenVPN VPC Address

In ethertype IPv4 (0x0800), length 76: 192.168.1.2 > 10.10.0.1: ICMP echo request, id 1, seq 39, length 40
Out ethertype IPv4 (0x0800), length 76: 10.10.0.1 > 192.168.1.2: ICMP echo reply, id 1, seq 39, length 40

Ping to OpenVPN VPN Address

In ethertype IPv4 (0x0800), length 76: 192.168.1.2 > 172.22.0.1: ICMP echo request, id 1, seq 43, length 40
Out ethertype IPv4 (0x0800), length 76: 172.22.0.1 > 192.168.1.2: ICMP echo reply, id 1, seq 43, length 40

Ping to Device on VPN

In ethertype IPv4 (0x0800), length 76: 192.168.1.2 > 172.22.22.154: ICMP echo request, id 1, seq 47, length 40
Out ethertype IPv4 (0x0800), length 76: 172.22.0.1 > 172.22.22.154: ICMP echo request, id 1, seq 47, length 40
In ethertype IPv4 (0x0800), length 76: 172.22.22.126 > 172.22.0.1: ICMP echo reply, id 1, seq 47, length 40
Out ethertype IPv4 (0x0800), length 76: 172.22.22.154 > 192.168.1.2: ICMP echo reply, id 1, seq 47, length 40

Basically the above is what we want to see when we swap to the FortiGate device.

Swapping the Meraki for the FortiGate:

Device on VPN (172.22.22.154/16)
|
|
OpenVPN Server (10.10.0.1/16 and 172.22.0.1/16)
|
|
Transit Gateway and Site-to-Site VPN
|
|
FORTIGATE 80F (192.168.7.254/24) <- Temp Dev network
|
|
Workstation (192.168.7.2/24) <- Temp Dev network

Using TCPDump on the OpenVPN Server:

Ping to OpenVPN VPC Address

In ethertype IPv4 (0x0800), length 76: 192.168.7.2 > 10.10.0.1: ICMP echo request, id 1, seq 617, length 40
Out ethertype IPv4 (0x0800), length 76: 10.10.0.1 > 192.168.7.2: ICMP echo reply, id 1, seq 617, length 40

When we Ping to OpenVPN VPN Address and Ping to Device on VPN the workstation gets "Request timed out." and TCPDump shows nothing.

These are the results of a debug flow:

On the FortiGate:

diagnose debug flow filter daddr 10.10.0.1
diagnose debug enable
diagnose debug flag trace start

On the workstation:

tracert 10.10.0.6

Result:

id=20085 trace_id=186 func=print_pkt_detail line=5824 msg="vd-root:0 received a packet(proto=1, 192.168.7.2:1->10.10.0.1:2048) tun_id=0.0.0.0 from internal. type=8, code=0, id=1, seq=639."
id=20085 trace_id=186 func=init_ip_session_common line=6003 msg="allocate a new session-00013f90, tun_id=0.0.0.0"
id=20085 trace_id=186 func=vf_ip_route_input_common line=2604 msg="find a route: flag=04000000 gw-XX.XX.XX.XX via AWS-WAN2-TUN1"
id=20085 trace_id=186 func=fw_forward_handler line=874 msg="Allowed by Policy-12:"

On the FortiGate:

diagnose debug flow filter daddr 172.22.0.1
diagnose debug flag trace start

On the Workstation:

tracert 172.22.0.1

Result:

id=20085 trace_id=185 func=print_pkt_detail line=5824 msg="vd-root:0 received a packet(proto=1, 192.168.7.2:1->172.22.0.1:2048) tun_id=0.0.0.0 from internal. type=8, code=0, id=1, seq=630."
id=20085 trace_id=185 func=init_ip_session_common line=6003 msg="allocate a new session-00013e7a, tun_id=0.0.0.0"
id=20085 trace_id=185 func=vf_ip_route_input_common line=2604 msg="find a route: flag=04000000 gw-XX.XX.XX.XX via AWS-WAN2-TUN1"
id=20085 trace_id=185 func=fw_forward_handler line=874 msg="Allowed by Policy-12:"

• It appears that the FortiGate is routing the traffic to the Site-to-Site VPN correctly now.
• The routes and security groups in AWS are correct because:
  • Meraki Traffic works successfully
  • FortiGate Traffic makes it to the OpenVPN Server 10.10.0.1

We have reviewed the iptables rules on the OpenVPN server and these are the relevant rules for these tests (192.168.7.0/24 is within 192.168.6.0/23):

## 192.168.6.0/23 ##
# ACCEPT ICMP ping to tun0 172.22.0.0/16
iptables -A FORWARD -i eth0 -s 192.168.6.0/23 -p icmp -o tun0 -d 172.22.0.0/16 -j ACCEPT
# ACCEPT already established connections from tun0 172.22.0.0/16
iptables -A FORWARD -m state -i tun0 -s 172.22.0.0/16 -o eth0 -d 192.168.6.0/23 --state ESTABLISHED,RELATED -j ACCEPT
# MASQUERADE Forwards to tun0 172.22.0.0/16 to avoid the need to push a route
iptables -t nat -A POSTROUTING -s 192.168.6.0/23 -o tun0 -d 172.22.0.0/16 -j MASQUERADE

[night_admin]

Thank you for the direction this gave me a bit to work with.

I set up the 80F in a dev environment. It has just wan2 connected and only 1 tunnel to AWS.

Looking at the flow log there was a reference to "Match policy routing id=..." for traffic to the 172.22.0.0/16. This was an SD-WAN Rule telling some networks to favor wan1. I have removed these rules for now but it is something that we want to work out.

id=20085 trace_id=55 func=print_pkt_detail line=5824 msg="vd-root:0 received a packet(proto=1, 192.168.7.2:1->172.22.22.154:2048) tun_id=0.0.0.0 from internal. type=8, code=0, id=1, seq=144."
id=20085 trace_id=55 func=init_ip_session_common line=6003 msg="allocate a new session-00006038, tun_id=0.0.0.0"
id=20085 trace_id=55 func=rpdb_srv_match_input line=1028 msg="Match policy routing id=2133393410: to 172.22.22.154 via ifindex-6"
id=20085 trace_id=55 func=vf_ip_route_input_common line=2604 msg="find a route: flag=04000000 gw-XX.XX.XX.XX via wan2"

Using this map as a summary of the map in the original post:

Device on VPN (172.22.22.154/16)
|
|
OpenVPN Server (10.10.0.1/16 and 172.22.0.1/16)
|
|
Transit Gateway and Site-to-Site VPN
|
|
Meraki (192.168.1.254/24)
|
|
Router/L3 Switch (192.168.1.1/24)
|
|
Workstation (192.168.1.2/24)

Using TCPDump on the OpenVPN Server:

Ping to OpenVPN Server VPC Address

In ethertype IPv4 (0x0800), length 76: 192.168.1.2 > 10.10.0.1: ICMP echo request, id 1, seq 39, length 40
Out ethertype IPv4 (0x0800), length 76: 10.10.0.1 > 192.168.1.2: ICMP echo reply, id 1, seq 39, length 40

Ping to OpenVPN Server VPN Address

In ethertype IPv4 (0x0800), length 76: 192.168.1.2 > 172.22.0.1: ICMP echo request, id 1, seq 43, length 40
Out ethertype IPv4 (0x0800), length 76: 172.22.0.1 > 192.168.1.2: ICMP echo reply, id 1, seq 43, length 40

Ping to Device on VPN

In ethertype IPv4 (0x0800), length 76: 192.168.1.2 > 172.22.22.154: ICMP echo request, id 1, seq 47, length 40
Out ethertype IPv4 (0x0800), length 76: 172.22.0.1 > 172.22.22.154: ICMP echo request, id 1, seq 47, length 40
In ethertype IPv4 (0x0800), length 76: 172.22.22.154 > 172.22.0.1: ICMP echo reply, id 1, seq 47, length 40
Out ethertype IPv4 (0x0800), length 76: 172.22.22.154 > 192.168.1.2: ICMP echo reply, id 1, seq 47, length 40

Basically the above is what we want to see with the new device.

Then testing with the FortiGate in place:

Device on VPN (172.22.22.154/16)
|
|
OpenVPN Server (10.10.0.1/16 and 172.22.0.1/16)
|
|
Transit Gateway and Site-to-Site VPN
|
|
FORTIGATE 80F (192.168.7.254/24) <- Testing network
|
|
Workstation (192.168.7.2) <- Testing network

Using TCPDump on the OpenVPN Server:

Ping to OpenVPN VPC Address

In ethertype IPv4 (0x0800), length 76: 192.168.7.2 > 10.10.0.1: ICMP echo request, id 1, seq 617, length 40
Out ethertype IPv4 (0x0800), length 76: 10.10.0.1 > 192.168.7.2: ICMP echo reply, id 1, seq 617, length 40

When we Ping to OpenVPN VPN Address and Ping to Device on VPN the workstation gets "Request timed out." and TCPDump shows nothing.

These are the results of debug flow on the FortiGate:

On the FortiGate:

diagnose debug flow filter daddr 10.10.0.1
diagnose debug enable
diagnose debug flag trace start

On the Workstation:

tracert 10.10.0.1

Result:

id=20085 trace_id=186 func=print_pkt_detail line=5824 msg="vd-root:0 received a packet(proto=1, 192.168.7.2:1->10.10.0.1:2048) tun_id=0.0.0.0 from internal. type=8, code=0, id=1, seq=639."
id=20085 trace_id=186 func=init_ip_session_common line=6003 msg="allocate a new session-00013f90, tun_id=0.0.0.0"
id=20085 trace_id=186 func=vf_ip_route_input_common line=2604 msg="find a route: flag=04000000 gw-XX.XX.XX.XX via AWS-WAN2-TUN1"
id=20085 trace_id=186 func=fw_forward_handler line=874 msg="Allowed by Policy-12:"

On the FortiGate:

diagnose debug flow filter daddr 172.22.0.1
diagnose debug flag trace start

On the Workstation:

tracert 172.22.0.1

Result:

id=20085 trace_id=185 func=print_pkt_detail line=5824 msg="vd-root:0 received a packet(proto=1, 192.168.7.2:1->172.22.0.1:2048) tun_id=0.0.0.0 from internal. type=8, code=0, id=1, seq=630."
id=20085 trace_id=185 func=init_ip_session_common line=6003 msg="allocate a new session-00013e7a, tun_id=0.0.0.0"
id=20085 trace_id=185 func=vf_ip_route_input_common line=2604 msg="find a route: flag=04000000 gw-XX.XX.XX.XX via AWS-WAN2-TUN1"
id=20085 trace_id=185 func=fw_forward_handler line=874 msg="Allowed by Policy-12:"

• It appears, based on the flow that the traffic is being routed to the tunnel correctly
• The Routes and Security groups are correct because:
  • Existing Meraki traffic works
  • Traffic is making it to the OpenVPN server when using the FortiGate

We reviewed the iptables rules on the OpenVPN Server. These are the rules relevant to our tests:

## 192.168.6.0/23 ##
# ACCEPT ICMP ping to tun0 172.22.0.0/16
iptables -A FORWARD -i eth0 -s 192.168.6.0/23 -p icmp -o tun0 -d 172.22.0.0/16 -j ACCEPT
# ACCEPT already established connections from tun0 172.22.0.0/16
iptables -A FORWARD -m state -i tun0 -s 172.22.0.0/16 -o eth0 -d 192.168.6.0/23 --state ESTABLISHED,RELATED -j ACCEPT
# MASQUERADE Forwards to tun0 172.22.0.0/16 to avoid the need to push a route
iptables -t nat -A POSTROUTING -s 192.168.6.0/23 -o tun0 -d 172.22.0.0/16 -j MASQUERADE

[night_admin]

Thank you for the direction this gave me a bit to work with.

I set up the 80F in a dev environment. It has just wan2 connected and only 1 tunnel to AWS.

Looking at the flow log there was a reference to "Match policy routing id=..." for traffic to the 172.22.0.0/16 network. This was an SD-WAN Rule telling some networks to favor wan1. I have removed these rules for now but it is something that we want to work out.

id=20085 trace_id=55 func=print_pkt_detail line=5824 msg="vd-root:0 received a packet(proto=1, 192.168.7.2:1->172.22.22.154:2048) tun_id=0.0.0.0 from internal. type=8, code=0, id=1, seq=144."
id=20085 trace_id=55 func=init_ip_session_common line=6003 msg="allocate a new session-00006038, tun_id=0.0.0.0"
id=20085 trace_id=55 func=rpdb_srv_match_input line=1028 msg="Match policy routing id=2133393410: to 172.22.22.154 via ifindex-6"
id=20085 trace_id=55 func=vf_ip_route_input_common line=2604 msg="find a route: flag=04000000 gw-XX.XX.XX.XX via wan2"

Using this map as a summary of the map in the original post

Device on VPN (172.22.22.154/16)
|
|
OpenVPN Server (10.10.0.1/16 and 172.22.0.1/16)
|
|
Transit Gateway and Site-to-Site VPN
|
|
Meraki (192.168.1.254/24)
|
|
Router/L3 Switch (192.168.1.1/24)
|
|
Workstation (192.168.1.2/24)

[vvarangoulis]

Hello,

Based on your last reply, it looks also you have configured SD-WAN in your FGT.
have you looked at the KB below, it might have some useful info for your case

https://community.fortinet.com/t5/FortiGate/Technical-Tip-Configure-FortiGate-SD-WAN-with-an-IPSEC-V...