FortiGate
FortiGate Next Generation Firewall utilizes purpose-built security processors and threat intelligence security services from FortiGuard labs to deliver top-rated protection and high performance, including encrypted traffic.
amalsky
Staff
Staff
Article Id 423036
Description This article describes how to configure, diagnose, and troubleshoot an IKEv2 site-to-site IPsec VPN between a Cisco router and a FortiGate firewall using a route-based (VTI) design.
It also documents the whole troubleshooting process with real command-line outputs, highlighting the differences between a non-working and a working deployment.
Scope FortiGate 7.2.x to 7.4.x, and 7.4.x to 7.6.x.
Solution Untitled Diagram.drawio.png

 

FortiGate IKEv2 implementation overview:

 

FortiGate implements a standards-compliant IKEv2 stack aligned with RFC 7296.
Route-based IPsec creates a logical tunnel interface that behaves as a routed interface and requires a /32 IP address.  

 

Cisco FlexVPN Smart Defaults – Initial Behavior:

 

Cisco FlexVPN Smart Defaults automatically negotiate IKEv2 and IPsec parameters.
In Cisco-to-Cisco deployments, this often results in transport-mode ESP or GRE-protected traffic that does not align with FortiGate's route-based expectations.

 

Troubleshooting Logic Overview:

 

The troubleshooting process follows this strict order:

  1. Verify Phase 1 (IKE SA).
  2. Verify Phase 2 (IPsec / Child SA).
  3. Verify data-plane traffic using counters and packet captures.
  4. Compare working vs non-working indicators at each step.
  5. Phase 1 Diagnostics – IKE SA (Working vs Non-Working).

 

Phase 1 diagnostics must always be validated first, as Phase 2 negotiation cannot occur unless the IKE SA is fully established.

 

FortiGate command:

 

High-level tunnel status (FortiGate).

 

Command:

 

get vpn ipsec tunnel summary

 

Working scenario output:

 

'CISCO-FLEXVPN' 192.168.12.2:0 selectors(total,up): 1/1 rx(pkt,err): 28/0 tx(pkt,err): 28/0

 

Interpretation (working):

  • Selectors 1/1 indicate Phase 2 selectors are installed.
  • RX/TX counters increment, confirming traffic flow.
  • Phases 1 and 2 are both operational.

 

Non-working scenario (typical):

 

'CISCO-FLEXVPN' 192.168.12.2:0 selectors(total,up): 1/0 rx(pkt,err): 0/0 tx(pkt,err): 0/0

 

Interpretation (non-working):

  • Selectors 1/0 indicate Phase 2 is not up.
  • No traffic is flowing.
  • Phase 1 must be checked in detail.

 

Cisco command:


show crypto ikev2 sa

 

Working scenario output:

 

Tunnel-id Local Remote Status
2 192.168.12.2/500 192.168.12.1/500 READY
Encr: AES-CBC-256, PRF: SHA512, DH Grp:5, Auth: PSK

 

Interpretation (working):

  • Status READY confirms Phase 1 (IKE SA) is established.
  • Proposals, Diffie-Hellman group, and authentication are correct.
  • Phase 2 negotiation is allowed to start.

 

Non-working scenario (typical indicators):

  • Status CONNECTING.
  • Repeated IKE_SA_INIT attempts.
  • No IPsec SA created.

 

Interpretation (non-working):

  • Phase 1 is not established.
  • Phase 2 cannot be negotiated.
  • Troubleshooting must focus on underlay reachability, IKE proposals, and authentication.

 

Phase 2 Diagnostics – IPsec / Child SA (initial failure).

 

FortiGate command:

 

diagnose vpn tunnel list

 

Non-working output excerpt:


status=up
child_num=0
accept_traffic=0
rxp=0 txp=0

 

Interpretation:

  • IKE SA exists, but no Child SA is installed.
  • FortiGate cannot forward traffic.
  • Phase 2 parameters or encapsulation mode are incorrect.

 

Packet Capture Analysis – Non-Working State.

WAN-Side Capture (Non-Working).

 

FortiGate command:

 

diagnose sniffer packet port1 "host 192.168.12.2 and (udp port 500 or udp port 4500 or esp)" 4 0 a

 

Observed output:


192.168.12.1.500 -> 192.168.12.2.500: udp
192.168.12.2.500 -> 192.168.12.1.500: udp

 

Interpretation:

  • Only IKE negotiation traffic (UDP/500) is observed.
  • No ESP packets are exchanged.
  • Confirms Phase 2 is not established.
  • Tunnel Interface Capture (Non-Working).

 

FortiGate command:

 

diagnose sniffer packet CISCO-FLEXVPN "icmp" 4 0 a

 

Observed output: 0 packets received by filter.

Interpretation:

  • No decrypted traffic reaches the tunnel interface.
  • Confirms absence of a working child SA.

 

Root cause identification.

 

The root cause was identified as:

  • Cisco FlexVPN Smart Defaults negotiating transport-mode ESP or GRE-style behavior.
  • FortiGate route-based IPsec requiring tunnel-mode ESP.
  • Phase 2 encapsulation and selectors not matching routed traffic.

 

The following adjustments were required.

 

Cisco:

 

  • Configure a Virtual Tunnel Interface (VTI).
  • Explicitly define a tunnel-mode IPsec transform-set.
  • Apply a custom IPsec profile to Tunnel0.

 

FortiGate:

 

  • Recreate Phase 2 using tunnel mode with broad selectors.
  • Assign a /32 IP address to the tunnel interface.
  • Configure static routes for the remote tunnel IP and remote LAN.

 

Phase 2 Verification – working state.

 

FortiGate command:

 

diagnose vpn tunnel list

 

Working output example

 

status=up
accept_traffic=1
rxp=18 txp=18
dec:pkts/bytes=18/1752
enc:pkts/bytes=18/2976

 

Interpretation:

  • Child SA is installed.
  • Traffic is encrypted and decrypted successfully.
  • Data plane is operational.

 

Packet Capture Analysis – working state.

WAN-Side Capture (Working).

 

FortiGate command:

 

diagnose sniffer packet port1 "host 192.168.12.2 and esp" 4 0 a

 

Observed output:

 

192.168.12.2 -> 192.168.12.1: ESP
192.168.12.1 -> 192.168.12.2: ESP

 

Interpretation:

  • ESP packets confirm Phase 2 encryption is active.

Tunnel Interface Capture (Working).

 

FortiGate command:

 

diagnose sniffer packet CISCO-FLEXVPN "icmp" 4 0 a

 

Observed output:

 

172.16.12.2 -> 172.16.12.1: icmp echo request
172.16.12.1 -> 172.16.12.2: icmp echo reply

 

Interpretation:

  • Traffic is decrypted correctly and routed via the tunnel interface.

 

Connectivity verification:

 

Cisco:

 

ping 172.16.12.1 source 172.16.12.2
Result: Success rate is 100 percent.

 

FortiGate:

 

execute ping 172.16.12.2
Result: 0 percent packet loss

 

Both directions succeed.

 

Interoperability notes:

  • Cisco FlexVPN Smart Defaults are optimised for Cisco-to-Cisco use cases.
  • FortiGate does not terminate GRE and expects tunnel-mode ESP.
  • IKE Phase 1 success does not guarantee data-plane success.
  • CHILD_SA presence and ESP visibility are critical indicators.
  • Packet sniffing clearly shows where traffic stops.
  • /30 on Cisco and /32 on FortiGate tunnel interfaces is a valid and supported design when routing is explicit.

 

Conclusion:
By forcing Cisco to use a VTI with tunnel mode ESP and aligning FortiGate's Phase 2 behavior, full IKEv2 interoperability is achieved.