Fortigate 200D (v5.6.2) IPSec VPN to AWS. Traffic makes it to EC2 but not back.

[cgbaroudi]

Hi,

I have 2 VPNs UP with AWS VPC. The VPNs are configured with failover so if one goes down the other picks up. Below we can see one VPN is up:

34.232.188.255 169.254.47.44/30 DOWN52.86.222.130 169.254.44.184/30 UP 

Info:

1. When I ping from my Fortigate to the EC2, I can see that the EC2 receives the ping request and sends a reply. The reply however never makes it back through the tunnel:tcpdump output on the AWS EC2:12:50:58.923392 IP 169.254.44.186 > 192.168.241.142: ICMP echo request, id 2769612:50:58.923405 IP 192.168.241.142 > 169.254.44.186: ICMP echo reply, id 276963. 2. From the fortigate I can ping both ends of the tunnel which is obvious from the time in ms: execute ping 169.254.44.185

PING 169.254.44.185 (169.254.44.185): 56 data bytes64 bytes from 169.254.44.185: icmp_seq=0 ttl=254 time=13.0 ms64 bytes from 169.254.44.185: icmp_seq=1 ttl=254 time=13.1 ms

execute ping 169.254.44.186

PING 169.254.44.186 (169.254.44.186): 56 data bytes64 bytes from 169.254.44.186: icmp_seq=0 ttl=255 time=0.0 ms64 bytes from 169.254.44.186: icmp_seq=1 ttl=255 time=0.0 ms4. 3. From the EC2 I cannot ping either end of the tunnel.5. SSH fails in the same way. 4. We see ssh session attempts and replies which do not make it back from the EC2:

06:00:30.829303 IP 192.168.241.142.ssh > 169.254.44.186.9589: Flags [S.], seq 2689521519, ack 2166390256, win 26847, options [mss 8961,sackOK,TS val 4354044 ecr 236943444,nop,wscale 7], length 0 06:00:38.633577 IP 169.254.44.186.9589 > 192.168.241.142.ssh: Flags , seq 2166390255, win 5272, options [mss 1318,sackOK,TS val 236945544 ecr 0,nop,wscale 2], length 0

Seems that traffic from the EC2 back to the Fortigate is being blocked yet debug flow on the fortigate shows no attempts or denial:

# execute ssh 192.168.241.142 id=20085 trace_id=2273 func=print_pkt_detail line=5293 msg="vd-root received a packet(proto=6, 169.254.44.186:9589->192.168.241.142:22) from local. flag , seq 2166390255, ack 0, win 5272" id=20085 trace_id=2273 func=init_ip_session_common line=5449 msg="allocate a new session-01a8007c" id=20085 trace_id=2273 func=iprope_dnat_check line=4754 msg="in-[], out-[vpn-23766442-1]" id=20085 trace_id=2273 func=iprope_dnat_tree_check line=835 msg="len=0" id=20085 trace_id=2273 func=iprope_dnat_check line=4767 msg="result: skb_flags-00000000, vid-0, ret-no-match, act-accept, flag-00000000" id=20085 trace_id=2273 func=ipsecdev_hard_start_xmit line=144 msg="enter IPsec interface-vpn-23766442-1" id=20085 trace_id=2273 func=esp_output4 line=1174 msg="IPsec encrypt/auth" id=20085 trace_id=2273 func=ipsec_output_finish line=534 msg="send to 66.162.199.129 via intf-wan1"

Any ideas would be very helpful. Please!!!

Thank you and Best Regards.

[Paul_S]

Maybe a mismatch of IPSEC encryption domain? NAT issue?  Maybe we need more info about your setup.

is this line related to the VPN?

id=20085 trace_id=2273 func=ipsec_output_finish line=534 msg="send to 66.162.199.129 via intf-wan1"

is 192.168.241.142 behind WAN1?

"enter IPsec interface-vpn-23766442-1"

shouldn't the action be leaving the tunnel?

[cgbaroudi]

Hi Paul,

Thank you for the response. I will answer your questions first then follow up with the complete CLI configuration.

1. 192.168.241.142 is the AWS VPC subnet

2. In the log you referred to I have "diag flow" enabled and filtering on one of the VPN endpoints (169.254.44.186 & 185)

3. From the log below, I understand that the fortigate figured that the ssh command to  192.168.241.142  is to be routed to the VPN interface vpn-23766442-1 then it says sending through the physical default route of  66.162.199.129 thru wan1. Am I correct? The follow up post has the full diag flow trace logs for ping and ssh.

Log:

id=20085 trace_id=2273 func=ipsecdev_hard_start_xmit line=144 msg="enter IPsec interface-vpn-23766442-1" id=20085 trace_id=2273 func=esp_output4 line=1174 msg="IPsec encrypt/auth" id=20085 trace_id=2273 func=ipsec_output_finish line=534 msg="send to 66.162.199.129 via intf-wan1"

Complete CLI config on fortigate follows. (version 5.6.2). Please note I made changes to update the setting to work with v5.6.2:

! Amazon Web Services ! Virtual Private Cloud

! AWS utilizes unique identifiers to manipulate the configuration of ! a VPN Connection. Each VPN Connection is assigned an identifier and is ! associated with two other identifiers, namely the ! Customer Gateway Identifier and Virtual Private Gateway Identifier. ! ! Your VPN Connection ID : vpn-23766442 ! Your Virtual Private Gateway ID : vgw-28da3041 ! Your Customer Gateway ID : cgw-3720ca5e ! ! ! This configuration consists of two tunnels. Both tunnels must be ! configured on your Customer Gateway. ! ! -------------------------------------------------------------------------------- ! IPSec Tunnel #1 ! -------------------------------------------------------------------------------- ! #1: Internet Key Exchange (IKE) Configuration ! ! A policy is established for the supported ISAKMP encryption, ! authentication, Diffie-Hellman, lifetime, and key parameters. ! Please note, these sample configurations are for the minimum requirement of AES128, SHA1, and DH Group 2. ! You will need to modify these sample configuration files to take advantage of AES256, SHA256, or other DH groups like 2, 14-18, 22, 23, and 24. ! ! The address of the external interface for your customer gateway must be a static address. ! Your customer gateway may reside behind a device performing network address translation (NAT). ! To ensure that NAT traversal (NAT-T) can function, you must adjust your firewall rules to unblock UDP port 4500. If not behind NAT, we recommend disabling NAT-T. ! ! Configuration begins in root VDOM.

config vpn ipsec phase1-interface edit vpn-23766442-0 ! Name must be shorter than 15 chars, best if shorter than 12 set interface "wan1"

! The IPSec Dead Peer Detection causes periodic messages to be ! sent to ensure a Security Association remains operational

set dpd enable set local-gw 66.x.x.x set dhgrp 2 set proposal aes128-sha1 set keylife 28800 set remote-gw 34.232.188.255 set psksecret pV5rbsXlyhWYLWEXB.gkmS56nMktjEMB set dpd-retryinterval 10 next end

! #2: IPSec Configuration ! ! The IPSec transform set defines the encryption, authentication, and IPSec ! mode parameters. ! ! Please note, you may use these additionally supported IPSec parameters for encryption like AES256 and other DH groups like 2, 5, 14-18, 22, 23, and 24.

config vpn ipsec phase2-interface edit "vpn-23766442-0" set phase1name "vpn-23766442-0" set proposal aes128-sha1 set dhgrp 2 set pfs enable set keylifeseconds 3600 next

! -------------------------------------------------------------------------------- ! #3: Tunnel Interface Configuration ! ! A tunnel interface is configured to be the logical interface associated ! with the tunnel. All traffic routed to the tunnel interface will be ! encrypted and transmitted to the VPC. Similarly, traffic from the VPC ! will be logically received on this interface. ! ! ! The address of the interface is configured with the setup for your ! Customer Gateway. If the address changes, the Customer Gateway and VPN ! Connection must be recreated with Amazon VPC. ! Perform this from the Global VDOM.

config global config system interface edit "vpn-23766442-0" set vdom "root" set ip 169.254.47.46 255.255.255.255 set allowaccess ping set type tunnel

! This option causes the router to reduce the Maximum Segment Size of ! TCP packets to prevent packet fragmentation. ! set tcp-mss 1379 set remote-ip 169.254.47.45 ??? set mtu 1427 set interface "wan1" next

! ---------------------------------------------------------------------------- ! #4 Static Route Configuration ! ! Your Customer Gateway needs to set a static route for the prefix corresponding to your ! VPC to send traffic over the tunnel interface. ! An example for a VPC with the prefix 10.0.0.0/16 is provided below: ! ! Configuration begins in root VDOM. ! First obtain the numbered list of static routes in use:

config vdom edit root show router static

! This will have entries for edit 1, edit 2, etc., showing the device, destination, and gateway for each static route configured. You need to use an unused routing entry for the static route into your VPC. So for example if there were routes 1, 2, and 3, then you would edit 4 below:

config router static edit 4 set device "vpn-23766442-0" set dst 192.168.241.142 255.255.0.0 next end

! Static routing does not allow for failover of traffic between tunnels. If there is a problem with one of the ! tunnels, we would want to failover the traffic to the second tunnel. This is done by using "gwdetect" in fortigate. ! The gwdetect command will ping the other end of the tunnel to check if the tunnel is up. If the pings fail, it will ! remove the static route from the routing table, and the second route in the table will become active. ! ! The following config will tell the Fortigate device what IP it should ping to test the tunnel. This IP should be ! the inside IP address of the virtual private gateway.

FMU-FW-02 # config system link-monitor

FMU-FW-02 (link-monitor) # edit 1 new entry '1' added

FMU-FW-02 (1) # set srcintf "vpn-23766442-0"

FMU-FW-02 (1) # set server "169.254.47.45"

FMU-FW-02 (1) # set interval 2

FMU-FW-02 (1) # set failtime 5

FMU-FW-02 (1) # next

FMU-FW-02 (link-monitor) # end

FMU-FW-02 # config system link-monitor

FMU-FW-02 (link-monitor) # show config system link-monitor edit "1" set srcintf "vpn-23766442-0" set server "169.254.47.45" set interval 2 next end

! ---------------------------------------------------------------------------- ! #5 Firewall Policy Configuration ! ! Create a firewall policy permitting traffic from your local subnet to the VPC subnet and vice versa ! ! This example policy permits all traffic from the local subnet to the VPC ! First, find the policies that exist ! Configuration begins in root VDOM.

config vdom edit root show firewall policy

! Next, create a new firewall policy starting with the next available policy ID. If policies 1, 2, 3, and 4 were shown, then in this example the policy created starts at 5

config firewall policy edit 5 set srcintf "vpn-23766442-0" set dstintf "internal" ! This is the interface out which your local LAN resides set srcaddr all set dstaddr all set action accept set schedule always set service ALL next end

! Now create a policy to permit traffic going the other way

config firewall policy edit 6 set srcintf internal ! This is the interface out which your local LAN resides set dstintf "vpn-23766442-0" set srcaddr all set dstaddr all set action accept set schedule always set service ALL next end

! -------------------------------------------------------------------------------- ! IPSec Tunnel #2 ! -------------------------------------------------------------------------------- ! #1: Internet Key Exchange (IKE) Configuration ! ! A policy is established for the supported ISAKMP encryption, ! authentication, Diffie-Hellman, lifetime, and key parameters. ! Please note, these sample configurations are for the minimum requirement of AES128, SHA1, and DH Group 2. ! You will need to modify these sample configuration files to take advantage of AES256, SHA256, or other DH groups like 2, 14-18, 22, 23, and 24. ! ! The address of the external interface for your customer gateway must be a static address. ! Your customer gateway may reside behind a device performing network address translation (NAT). ! To ensure that NAT traversal (NAT-T) can function, you must adjust your firewall rules to unblock UDP port 4500. If not behind NAT, we recommend disabling NAT-T. ! ! Configuration begins in root VDOM.

config vpn ipsec phase1-interface edit vpn-23766442-1 ! Name must be shorter than 15 chars, best if shorter than 12 set interface "wan1"

! The IPSec Dead Peer Detection causes periodic messages to be ! sent to ensure a Security Association remains operational

set dpd enable ### set dpd on-demand set local-gw 66.x.x.x set dhgrp 2 set proposal aes128-sha1 set keylife 28800 set remote-gw 52.86.222.130 set psksecret k3Z8Q8K1L3MUClD1MbaPQNNSrffFRS5K set dpd-retryinterval 10 next

### end

! #2: IPSec Configuration ! ! The IPSec transform set defines the encryption, authentication, and IPSec ! mode parameters. ! ! Please note, you may use these additionally supported IPSec parameters for encryption like AES256 and other DH groups like 2, 5, 14-18, 22, 23, and 24.

config vpn ipsec phase2-interface edit "vpn-23766442-1" set phase1name "vpn-23766442-1" set proposal aes128-sha1 set dhgrp 2 set pfs enable set keylifeseconds 3600 next end

! -------------------------------------------------------------------------------- ! #3: Tunnel Interface Configuration ! ! A tunnel interface is configured to be the logical interface associated ! with the tunnel. All traffic routed to the tunnel interface will be ! encrypted and transmitted to the VPC. Similarly, traffic from the VPC ! will be logically received on this interface. ! ! ! The address of the interface is configured with the setup for your ! Customer Gateway. If the address changes, the Customer Gateway and VPN ! Connection must be recreated with Amazon VPC. ! Perform this from the Global VDOM.

config global config system interface edit "vpn-23766442-1" set vdom "root" set ip 169.254.44.186 255.255.255.255 set allowaccess ping set type tunnel

! This option causes the router to reduce the Maximum Segment Size of ! TCP packets to prevent packet fragmentation. ! set tcp-mss 1379 set remote-ip 169.254.44.185 ??? set mtu 1427 set interface "wan1" next

### endconfig

! ---------------------------------------------------------------------------- ! #4 Static Route Configuration ! ! Your Customer Gateway needs to set a static route for the prefix corresponding to your ! VPC to send traffic over the tunnel interface. ! An example for a VPC with the prefix 10.0.0.0/16 is provided below: ! ! Configuration begins in root VDOM. ! First obtain the numbered list of static routes in use:

### below is only for listing static routes. Syntax does not work on v6.3

config vdom edit root show router static

! This will have entries for edit 1, edit 2, etc., showing the device, destination, and gateway for each static route configured. You need to use an unused routing entry for the static route into your VPC. So for example if there were routes 1, 2, and 3, then you would edit 4 below:

config router static edit 4 set device "vpn-23766442-1" set dst 192.168.0.0 255.255.0.0 next end

! Static routing does not allow for failover of traffic between tunnels. If there is a problem with one of the ! tunnels, we would want to failover the traffic to the second tunnel. This is done by using "gwdetect" in fortigate. ! The gwdetect command will ping the other end of the tunnel to check if the tunnel is up. If the pings fail, it will ! remove the static route from the routing table, and the second route in the table will become active. ! ! The following config will tell the Fortigate device what IP it should ping to test the tunnel. This IP should be ! the inside IP address of the virtual private gateway.

### Version 6.3:

FMU-FW-02 # config system link-monitor

FMU-FW-02 (link-monitor) # edit 2 new entry '1' added

FMU-FW-02 (1) # set srcintf "vpn-23766442-1"

FMU-FW-02 (1) # set server "169.254.44.185"

FMU-FW-02 (1) # set interval 2

FMU-FW-02 (1) # set failtime 5

FMU-FW-02 (1) # next

FMU-FW-02 (link-monitor) # end

### Version 5.x

config vdom edit root config router gwdetect edit 1 set interface "vpn-23766442-1" set server "169.254.44.185" ! Using the following command, you can set the interval and failtime for gwdetect. Interval is number of seconds ! between pings. Failtime is the number of lost consecutive pings.Using the respective values of 2 and 5, your tunnel ! will failover in 10 seconds. ! set interval 2 set failtime 5 next end

! ---------------------------------------------------------------------------- ! #5 Firewall Policy Configuration ! ! Create a firewall policy permitting traffic from your local subnet to the VPC subnet and vice versa ! ! This example policy permits all traffic from the local subnet to the VPC ! First, find the policies that exist ! Configuration begins in root VDOM.

config vdom edit root show firewall policy

! Next, create a new firewall policy starting with the next available policy ID. If policies 1, 2, 3, and 4 were shown, then in this example the policy created starts at 5

config firewall policy edit 5 set srcintf "vpn-23766442-1" set dstintf "internal" ! This is the interface out which your local LAN resides set srcaddr all set dstaddr all set action accept set schedule always set service ALL next end

! Now create a policy to permit traffic going the other way

config firewall policy edit 6 set srcintf internal ! This is the interface out which your local LAN resides set dstintf "vpn-23766442-1" set srcaddr all set dstaddr all set action accept set schedule always set service ALL next end

[cgbaroudi]

Here are the full flow trace logs for ping and ssh from the local Fortigate to the AWS EC2 (Centos):

FMU-FW-02 # diag debug flow filter addr 192.168.241.142

FMU-FW-02 # diag debug flow trace start 100

FMU-FW-02 # execute ssh 192.168.241.142 id=20085 trace_id=2273 func=print_pkt_detail line=5293 msg="vd-root received a packet(proto=6, 169.254.44.186:9589->192.168.241.142:22) from local. fl id=20085 trace_id=2273 func=init_ip_session_common line=5449 msg="allocate a new session-01a8007c" id=20085 trace_id=2273 func=iprope_dnat_check line=4754 msg="in-[], out-[vpn-23766442-1]" id=20085 trace_id=2273 func=iprope_dnat_tree_check line=835 msg="len=0" id=20085 trace_id=2273 func=iprope_dnat_check line=4767 msg="result: skb_flags-00000000, vid-0, ret-no-match, act-accept, flag-00000000" id=20085 trace_id=2273 func=ipsecdev_hard_start_xmit line=144 msg="enter IPsec interface-vpn-23766442-1" id=20085 trace_id=2273 func=esp_output4 line=1174 msg="IPsec encrypt/auth" id=20085 trace_id=2273 func=ipsec_output_finish line=534 msg="send to 66.162.199.129 via intf-wan1"

id=20085 trace_id=2274 func=print_pkt_detail line=5293 msg="vd-root received a packet(proto=6, 169.254.44.186:9589->192.168.241.142:22) from local. fl id=20085 trace_id=2274 func=resolve_ip_tuple_fast line=5368 msg="Find an existing session, id-01a8007c, original direction" id=20085 trace_id=2274 func=ipsecdev_hard_start_xmit line=144 msg="enter IPsec interface-vpn-23766442-1" id=20085 trace_id=2274 func=esp_output4 line=1174 msg="IPsec encrypt/auth" id=20085 trace_id=2274 func=ipsec_output_finish line=534 msg="send to 66.162.199.129 via intf-wan1" id=20085 trace_id=2275 func=print_pkt_detail line=5293 msg="vd-root received a packet(proto=6, 169.254.44.186:9589->192.168.241.142:22) from local. fl id=20085 trace_id=2275 func=resolve_ip_tuple_fast line=5368 msg="Find an existing session, id-01a8007c, original direction" id=20085 trace_id=2275 func=ipsecdev_hard_start_xmit line=144 msg="enter IPsec interface-vpn-23766442-1" id=20085 trace_id=2275 func=esp_output4 line=1174 msg="IPsec encrypt/auth" id=20085 trace_id=2275 func=ipsec_output_finish line=534 msg="send to 66.162.199.129 via intf-wan1" id=20085 trace_id=2276 func=print_pkt_detail line=5293 msg="vd-root received a packet(proto=6, 169.254.44.186:9589->192.168.241.142:22) from local. fl id=20085 trace_id=2276 func=init_ip_session_common line=5449 msg="allocate a new session-01a8018e" id=20085 trace_id=2276 func=iprope_dnat_check line=4754 msg="in-[], out-[vpn-23766442-1]" id=20085 trace_id=2276 func=iprope_dnat_tree_check line=835 msg="len=0" id=20085 trace_id=2276 func=iprope_dnat_check line=4767 msg="result: skb_flags-00000000, vid-0, ret-no-match, act-accept, flag-00000000" id=20085 trace_id=2276 func=ipsecdev_hard_start_xmit line=144 msg="enter IPsec interface-vpn-23766442-1" id=20085 trace_id=2276 func=esp_output4 line=1174 msg="IPsec encrypt/auth" id=20085 trace_id=2276 func=ipsec_output_finish line=534 msg="send to 66.162.199.129 via intf-wan1" id=20085 trace_id=2277 func=print_pkt_detail line=5293 msg="vd-root received a packet(proto=6, 169.254.44.186:9589->192.168.241.142:22) from local. fl id=20085 trace_id=2277 func=init_ip_session_common line=5449 msg="allocate a new session-01a802e0" id=20085 trace_id=2277 func=iprope_dnat_check line=4754 msg="in-[], out-[vpn-23766442-1]" id=20085 trace_id=2277 func=iprope_dnat_tree_check line=835 msg="len=0" id=20085 trace_id=2277 func=iprope_dnat_check line=4767 msg="result: skb_flags-00000000, vid-0, ret-no-match, act-accept, flag-00000000" id=20085 trace_id=2277 func=ipsecdev_hard_start_xmit line=144 msg="enter IPsec interface-vpn-23766442-1" id=20085 trace_id=2277 func=esp_output4 line=1174 msg="IPsec encrypt/auth" id=20085 trace_id=2277 func=ipsec_output_finish line=534 msg="send to 66.162.199.129 via intf-wan1"

FMU-FW-02 # execute ping 192.168.241.142 PING 192.168.241.142 (192.168.241.142): 56 data bytes id=20085 trace_id=2278 func=print_pkt_detail line=5293 msg="vd-root received a packet(proto=1, 169.254.44.186:47277->192.168.241.142:2048) from local. id=20085 trace_id=2278 func=init_ip_session_common line=5449 msg="allocate a new session-01a83a1d" id=20085 trace_id=2278 func=iprope_dnat_check line=4754 msg="in-[], out-[vpn-23766442-1]" id=20085 trace_id=2278 func=iprope_dnat_tree_check line=835 msg="len=0" id=20085 trace_id=2278 func=iprope_dnat_check line=4767 msg="result: skb_flags-00000000, vid-0, ret-no-match, act-accept, flag-00000000" id=20085 trace_id=2278 func=ipsecdev_hard_start_xmit line=144 msg="enter IPsec interface-vpn-23766442-1" id=20085 trace_id=2278 func=esp_output4 line=1174 msg="IPsec encrypt/auth" id=20085 trace_id=2278 func=ipsec_output_finish line=534 msg="send to 66.162.199.129 via intf-wan1" id=20085 trace_id=2279 func=print_pkt_detail line=5293 msg="vd-root received a packet(proto=1, 169.254.44.186:47277->192.168.241.142:2048) from local. id=20085 trace_id=2279 func=resolve_ip_tuple_fast line=5368 msg="Find an existing session, id-01a83a1d, original direction" id=20085 trace_id=2279 func=ipsecdev_hard_start_xmit line=144 msg="enter IPsec interface-vpn-23766442-1" id=20085 trace_id=2279 func=esp_output4 line=1174 msg="IPsec encrypt/auth" id=20085 trace_id=2279 func=ipsec_output_finish line=534 msg="send to 66.162.199.129 via intf-wan1" id=20085 trace_id=2280 func=print_pkt_detail line=5293 msg="vd-root received a packet(proto=1, 169.254.44.186:47277->192.168.241.142:2048) from local. id=20085 trace_id=2280 func=resolve_ip_tuple_fast line=5368 msg="Find an existing session, id-01a83a1d, original direction" id=20085 trace_id=2280 func=ipsecdev_hard_start_xmit line=144 msg="enter IPsec interface-vpn-23766442-1" id=20085 trace_id=2280 func=esp_output4 line=1174 msg="IPsec encrypt/auth" id=20085 trace_id=2280 func=ipsec_output_finish line=534 msg="send to 66.162.199.129 via intf-wan1" id=20085 trace_id=2281 func=print_pkt_detail line=5293 msg="vd-root received a packet(proto=1, 169.254.44.186:47277->192.168.241.142:2048) from local. id=20085 trace_id=2281 func=resolve_ip_tuple_fast line=5368 msg="Find an existing session, id-01a83a1d, original direction" id=20085 trace_id=2281 func=ipsecdev_hard_start_xmit line=144 msg="enter IPsec interface-vpn-23766442-1" id=20085 trace_id=2281 func=esp_output4 line=1174 msg="IPsec encrypt/auth" id=20085 trace_id=2281 func=ipsec_output_finish line=534 msg="send to 66.162.199.129 via intf-wan1" id=20085 trace_id=2282 func=print_pkt_detail line=5293 msg="vd-root received a packet(proto=1, 169.254.44.186:47277->192.168.241.142:2048) from local. id=20085 trace_id=2282 func=resolve_ip_tuple_fast line=5368 msg="Find an existing session, id-01a83a1d, original direction" id=20085 trace_id=2282 func=ipsecdev_hard_start_xmit line=144 msg="enter IPsec interface-vpn-23766442-1" id=20085 trace_id=2282 func=esp_output4 line=1174 msg="IPsec encrypt/auth" id=20085 trace_id=2282 func=ipsec_output_finish line=534 msg="send to 66.162.199.129 via intf-wan1"

--- 192.168.241.142 ping statistics --- 5 packets transmitted, 0 packets received, 100% packet loss

[emnoc]

Sugestion

1>

if you have 2x VPN shutting one down and try your test

2>

bring the one shutdown back up and shutdown the other and retest

3>

run diag sniffer packet <vpn interface name> "icmp" while doing your ping test

4>

Dump the route table for the  VPN-tunnels

e.g

get router info routing all | gre vpn

Ken

[cgbaroudi]

Hi Ken,

Thank you. 1> if you have 2x VPN shutting one down and try your test

Answer: One VPN was already shut down.   2> bring the one shutdown back up and shutdown the other and retest Answer: Done and got the same results.   3> run diag sniffer packet <vpn interface name> "icmp" while doing your ping test Answer: The log from the diag sniffer command provided below. Note the heartbeat pings before and after the ping test. No replies from 192.168.241.142:

5.684973 169.254.47.46 -> 169.254.47.45: icmp: echo request 5.696837 169.254.47.45 -> 169.254.47.46: icmp: echo reply 5.828415 169.254.47.46 -> 192.168.241.142: icmp: echo request 6.828322 169.254.47.46 -> 192.168.241.142: icmp: echo request 7.694965 169.254.47.46 -> 169.254.47.45: icmp: echo request 7.706828 169.254.47.45 -> 169.254.47.46: icmp: echo reply 7.828316 169.254.47.46 -> 192.168.241.142: icmp: echo request 8.828312 169.254.47.46 -> 192.168.241.142: icmp: echo request 9.704955 169.254.47.46 -> 169.254.47.45: icmp: echo request 9.716750 169.254.47.45 -> 169.254.47.46: icmp: echo reply 9.828297 169.254.47.46 -> 192.168.241.142: icmp: echo request 11.714945 169.254.47.46 -> 169.254.47.45: icmp: echo request 11.726765 169.254.47.45 -> 169.254.47.46: icmp: echo reply

  4> Dump the route table for the  VPN-tunnels

# get router info routing-table all | grep vpn S 10.108.0.0/16 [10/0] via 169.254.34.113, vpn-2574934c-0 S 10.134.8.0/21 [10/0] via 169.254.14.161, vpn-f95d47eb-0 C 169.254.14.53/32 is directly connected, vpn-f95d47eb-1 C 169.254.14.54/32 is directly connected, vpn-f95d47eb-1 C 169.254.14.161/32 is directly connected, vpn-f95d47eb-0 C 169.254.14.162/32 is directly connected, vpn-f95d47eb-0 C 169.254.34.17/32 is directly connected, vpn-2574934c-1 C 169.254.34.18/32 is directly connected, vpn-2574934c-1 C 169.254.34.113/32 is directly connected, vpn-2574934c-0 C 169.254.34.114/32 is directly connected, vpn-2574934c-0 C 169.254.47.45/32 is directly connected, vpn-23766442-0 C 169.254.47.46/32 is directly connected, vpn-23766442-0 S 192.168.0.0/16 [10/0] via 169.254.47.45, vpn-23766442-0

Thank you again.

Kind Regards.

[Paul_S]

these route table entries are the only one's for the VPN in question, yes? (vpn-23766442)?

C 169.254.47.45/32 is directly connected, vpn-23766442-0 C 169.254.47.46/32 is directly connected, vpn-23766442-0 S 192.168.0.0/16 [10/0] via 169.254.47.45, vpn-23766442-0

can you ping anything on the 192.168.0.0/16 network since that is on the far end of your VPN?

in your first post you said you see the ping reply, but it doesn't leave the tunnel. That means you see it on one interface, but it never leaves out another interface. what interface is the ping reply supposed to exit on?

[cgbaroudi]

Hi Paul,

Thank you.

Correct, the 3 routes you listed are the ones for the VPN in question.

I will rephrase perhaps more accurately. On the EC2 inside the AWS network I have tcpdump running. When I ping the AWS EC2 from the fortigate or even a PC from our local network, tcpdump will show the ICMP packet coming through the VPN to the EC2, then shows the EC2 responding but the reply never makes it back through the VPN. This also correlates with the fact that I can ping both endpoints of the VPN from the fortigate but I cannot ping either endpoint of the VPN from the AWS EC2 instance which incidentally is running Centos. It appears that something on the fortigate is blocking incoming traffic through the VPN interface while allowing outgoing traffic from the fortigate.

So to answer your second question, no I cannot ping the AWS instance from my local network from the fortigate or host. tcpdump shows me the ping is reaching the AWS EC2, the EC2 replies but the reply cannot get back through the VPN. Same with ssh traffic or any other traffic.

Can we trace dropped incoming VPN traffic? If we can and determine that the fortigate is not dropping incoming traffic then we can focus on AWS causing the issue.

Thank you again. I am very grateful for all your help.

[cgbaroudi]

Hey guys,

Thank you all for your help.

Problem was as suspected, static routes needed to be added to the VPN configuration which include:

a. static route for the VPN tunnel endpoints

b. static route for all our internal networks

Specifically in my case, the routes I had to add under AWS>VPC>VPN COnnections>Static Routes TAB are:

VPN endpoints:            169.254.47.44/30 Internal network 1:      10.20.10.0/24 Internal network 2:      10.96.0.0/16 Naturally, I can also ping the tunnel end point from the AWS EC2 instance.

Thank you all again (Paul S. and emnoc)

Kind Regards,

cgb

[cgbaroudi]

Hey guys,

Thank you all for your help.

The problem was as I suspected, static routes needed to be added to the VPN configuration which include:

a. static route for the VPN tunnel endpoints

b. static route for all our internal networks

Specifically, in my case, under AWS>VPC>VPN Connections>Static Routes Tab, I added:

VPN endpoints:            169.254.47.44/30 Internal network 1:      10.20.10.0/24 Internal network 2:      10.96.0.0/16 Naturally, I can now ping the tunnel endpoint from the AWS EC2 instance.

Thank you all again (Paul S. and emnoc)

Kind Regards,

cgb