Technical Tip: Configuring and troubleshooting an IKEv2 site-to-site VPN between Cisco FlexVPN with smart defaults and a FortiGate using VTI

FortiGate IKEv2 implementation overview:

FortiGate implements a standards-compliant IKEv2 stack aligned with RFC 7296.
Route-based IPsec creates a logical tunnel interface that behaves as a routed interface and requires a /32 IP address.  

Cisco FlexVPN Smart Defaults – Initial Behavior:

Cisco FlexVPN Smart Defaults automatically negotiate IKEv2 and IPsec parameters.
In Cisco-to-Cisco deployments, this often results in transport-mode ESP or GRE-protected traffic that does not align with FortiGate's route-based expectations.

Troubleshooting Logic Overview:

The troubleshooting process follows this strict order:

1. Verify Phase 1 (IKE SA).
2. Verify Phase 2 (IPsec / Child SA).
3. Verify data-plane traffic using counters and packet captures.
4. Compare working vs non-working indicators at each step.
5. Phase 1 Diagnostics – IKE SA (Working vs Non-Working).

Phase 1 diagnostics must always be validated first, as Phase 2 negotiation cannot occur unless the IKE SA is fully established.

FortiGate command:

High-level tunnel status (FortiGate).

Command:

get vpn ipsec tunnel summary

Working scenario output:

'CISCO-FLEXVPN' 192.168.12.2:0 selectors(total,up): 1/1 rx(pkt,err): 28/0 tx(pkt,err): 28/0

Interpretation (working):

• Selectors 1/1 indicate Phase 2 selectors are installed.
• RX/TX counters increment, confirming traffic flow.
• Phases 1 and 2 are both operational.

Non-working scenario (typical):

'CISCO-FLEXVPN' 192.168.12.2:0 selectors(total,up): 1/0 rx(pkt,err): 0/0 tx(pkt,err): 0/0

Interpretation (non-working):

• Selectors 1/0 indicate Phase 2 is not up.
• No traffic is flowing.
• Phase 1 must be checked in detail.

Cisco command:

show crypto ikev2 sa

Working scenario output:

Tunnel-id Local Remote Status
2 192.168.12.2/500 192.168.12.1/500 READY
Encr: AES-CBC-256, PRF: SHA512, DH Grp:5, Auth: PSK

Interpretation (working):

• Status READY confirms Phase 1 (IKE SA) is established.
• Proposals, Diffie-Hellman group, and authentication are correct.
• Phase 2 negotiation is allowed to start.

Non-working scenario (typical indicators):

• Status CONNECTING.
• Repeated IKE_SA_INIT attempts.
• No IPsec SA created.

Interpretation (non-working):

• Phase 1 is not established.
• Phase 2 cannot be negotiated.
• Troubleshooting must focus on underlay reachability, IKE proposals, and authentication.

Phase 2 Diagnostics – IPsec / Child SA (initial failure).

FortiGate command:

diagnose vpn tunnel list

Non-working output excerpt:

status=up
child_num=0
accept_traffic=0
rxp=0 txp=0

Interpretation:

• IKE SA exists, but no Child SA is installed.
• FortiGate cannot forward traffic.
• Phase 2 parameters or encapsulation mode are incorrect.

Packet Capture Analysis – Non-Working State.

WAN-Side Capture (Non-Working).

FortiGate command:

diagnose sniffer packet port1 "host 192.168.12.2 and (udp port 500 or udp port 4500 or esp)" 4 0 a

Observed output:

192.168.12.1.500 -> 192.168.12.2.500: udp
192.168.12.2.500 -> 192.168.12.1.500: udp

Interpretation:

• Only IKE negotiation traffic (UDP/500) is observed.
• No ESP packets are exchanged.
• Confirms Phase 2 is not established.
• Tunnel Interface Capture (Non-Working).

FortiGate command:

diagnose sniffer packet CISCO-FLEXVPN "icmp" 4 0 a

Observed output: 0 packets received by filter.

Interpretation:

• No decrypted traffic reaches the tunnel interface.
• Confirms absence of a working child SA.

Root cause identification.

The root cause was identified as:

• Cisco FlexVPN Smart Defaults negotiating transport-mode ESP or GRE-style behavior.
• FortiGate route-based IPsec requiring tunnel-mode ESP.
• Phase 2 encapsulation and selectors not matching routed traffic.

The following adjustments were required.

Cisco:

• Configure a Virtual Tunnel Interface (VTI).
• Explicitly define a tunnel-mode IPsec transform-set.
• Apply a custom IPsec profile to Tunnel0.

FortiGate:

• Recreate Phase 2 using tunnel mode with broad selectors.
• Assign a /32 IP address to the tunnel interface.
• Configure static routes for the remote tunnel IP and remote LAN.

Phase 2 Verification – working state.

FortiGate command:

diagnose vpn tunnel list

Working output example

status=up
accept_traffic=1
rxp=18 txp=18
dec:pkts/bytes=18/1752
enc:pkts/bytes=18/2976

Interpretation:

• Child SA is installed.
• Traffic is encrypted and decrypted successfully.
• Data plane is operational.

Packet Capture Analysis – working state.

WAN-Side Capture (Working).

FortiGate command:

diagnose sniffer packet port1 "host 192.168.12.2 and esp" 4 0 a

Observed output:

192.168.12.2 -> 192.168.12.1: ESP
192.168.12.1 -> 192.168.12.2: ESP

Interpretation:

• ESP packets confirm Phase 2 encryption is active.

Tunnel Interface Capture (Working).

FortiGate command:

diagnose sniffer packet CISCO-FLEXVPN "icmp" 4 0 a

Observed output:

172.16.12.2 -> 172.16.12.1: icmp echo request
172.16.12.1 -> 172.16.12.2: icmp echo reply

Interpretation:

• Traffic is decrypted correctly and routed via the tunnel interface.

Connectivity verification:

Cisco:

ping 172.16.12.1 source 172.16.12.2
Result: Success rate is 100 percent.

FortiGate:

execute ping 172.16.12.2
Result: 0 percent packet loss

Both directions succeed.

Interoperability notes:

• Cisco FlexVPN Smart Defaults are optimised for Cisco-to-Cisco use cases.
• FortiGate does not terminate GRE and expects tunnel-mode ESP.
• IKE Phase 1 success does not guarantee data-plane success.
• CHILD_SA presence and ESP visibility are critical indicators.
• Packet sniffing clearly shows where traffic stops.
• /30 on Cisco and /32 on FortiGate tunnel interfaces is a valid and supported design when routing is explicit.

Conclusion:
By forcing Cisco to use a VTI with tunnel mode ESP and aligning FortiGate's Phase 2 behavior, full IKEv2 interoperability is achieved.