Technical Tip: Troubleshooting IPsec tunnel with FortiGates in Hub-and-Spoke using Starlink ISP
| Description | This article describes how to troubleshoot when Hub-and-Spoke FortiGates are down by using Starlink ISP. |
| Scope | FortiGate, Hub-and-Spoke VPN, Starlink ISP. |
| Solution | When running packet capture between the Hub-and-Spoke, it may be observed that there is a two-way traffic between FortiGates at port 4500. Also, when checking the debug on the Spoke side, there is no error notification only the typical connection expiring due to phase1 down.
For IKE debugging troubleshoot commands, refer the following article: Troubleshooting Tip: IPsec VPN tunnels.
Note: The debugging results below come from enabling debugging in the article linked above.
Spoke:
ike V=root:0:Spoke-01: initiator: main mode get 1st response... ike V=root:0:Spoke-01: VID RFC 3947 4A131C81070358455C5728F20E95452F ike V=root:0:Spoke-01: VID DPD AFCAD71368A1F1C96B8696FC77570100 ike V=root:0:Spoke-01: DPD negotiated ike V=root:0:Spoke-01: VID FORTIGATE 8299031757A36082C6A621DE00000000 ike V=root:0:Spoke-01: peer is FortiGate/FortiOS (v0 b0) ike V=root:0:Spoke-01: VID FRAGMENTATION 4048B7D56EBCE88525E7DE7F00D6C2D3 ike V=root:0:Spoke-01: VID FRAGMENTATION 4048B7D56EBCE88525E7DE7F00D6C2D3C0000000 ike V=root:0:Spoke-01: VID Fortinet Auto-Discovery Sender 9B15E65A871AFF342666623BA5022E60 ike V=root:0:Spoke-01: VID Fortinet Exchange Interface IP A58FEC5036F57B21E8B499E336C76EE6 ike V=root:0:Spoke-01: selected NAT-T version: RFC 3947 ike V=root:0:Spoke-01: negotiation result ike V=root:0:Spoke-01: proposal id = 1: ike V=root:0:Spoke-01: protocol id = ISAKMP: ike V=root:0:Spoke-01: trans_id = KEY_IKE. ike V=root:0:Spoke-01: encapsulation = IKE/none ike V=root:0:Spoke-01: type=OAKLEY_ENCRYPT_ALG, val=AES_CBC, key-len=128 ike V=root:0:Spoke-01: type=OAKLEY_HASH_ALG, val=SHA2_256. ike V=root:0:Spoke-01: type=AUTH_METHOD, val=PRESHARED_KEY. ike V=root:0:Spoke-01: type=OAKLEY_GROUP, val=MODP2048. ike V=root:0:Spoke-01: ISAKMP SA lifetime=86400 ike V=root:0:Spoke-01: generate DH public value request queued .... ike V=root:0:Spoke-01: sent IKE msg (ident_i2send): 192.168.x.x:500->49.255.x.x:500, len=380, vrf=0, id=f5a048ff7b30e8ff/1ff8f8be29d1f74a ike V=root:0: comes 49.255.x.x:500->192.168.x.x:500,ifindex=3,vrf=0,len=380.... ike V=root:0: IKEv1 exchange=Identity Protection id=f5a048ff7b30e8ff/1ff8f8be29d1f74a len=380 vrf=0 .... ike V=root:0:Spoke-01: initiator: main mode get 2nd response... ike V=root:0:Spoke-01: received NAT-D payload type 20 ike V=root:0:Spoke-01: received NAT-D payload type 20 ike V=root:0:Spoke-01: NAT detected: ME ike V=root:0:Spoke-01: NAT-T float port 4500 .... ike V=root:0:Spoke-01: sent IKE msg (P1_RETRANSMIT): 192.168.x.x:4500->49.255.x.x:4500, len=140, vrf=0, id=f5a048ff7b30e8ff/1ff8f8be29d1f 74a .... ike V=root:0:Spoke-01: sent IKE msg (P1_RETRANSMIT): 192.168.x.x:4500->49.255.x.x:4500, len=140, vrf=0, id=f5a048ff7b30e8ff/1ff8f8be29d1f 74a ike :shrank heap by 159744 bytes ike V=root:0:Spoke-01: IPsec SA connect 3 192.168.x.x->49.255.x.x:0 ike V=root:0:Spoke-01: using existing connection ike V=root:0:Spoke-01: config found ike V=root:0:Spoke-01: IPsec SA connect 3 192.168.x.x->49.255.x.x:500 negotiating ike V=root:0:Spoke-01:Spoke-01:3: ISAKMP SA still negotiating, queuing quick-mode request .... ike V=root:0:Spoke-01: sent IKE msg (P1_RETRANSMIT): 192.168.x.x:4500->49.255.x.x:4500, len=140, vrf=0, id=f5a048ff7b30e8ff/1ff8f8be29d1f 74a ike V=root:0:Spoke-01: negotiation timeout, deleting ike V=root:0:Spoke-01: connection expiring due to phase1 down ike V=root:0:Spoke-01: going to be deleted ike V=root:0:Spoke-01: schedule auto-negotiate ike V=root:0:Spoke-01: auto-negotiate connection
On the other hand, the IKE debugging on the HUB shows the error 'error, received '' expected 'starlink''.
ike V=root:0: comes 65.181.15.2:5019->49.255.x.x:500,ifindex=5,vrf=0,len=612.... ike V=root:0: IKEv1 exchange=Identity Protection id=a88b04275081853f/0000000000000000 len=612 vrf=0 ike 0: in A88B04275081853F00000000000000000110020000000000000002640D000154000000010000000100000148010100080300002801010000800B000100 0C00040001518080010007800E008080030001800200048004000E0300002802010000800B0001000C00040001518080010007800E00808003000180020004800400 050300002803010000800B0001000C00040001518080010007800E010080030001800200048004000E0300002804010000800B0001000C0004000151808001000780 0E01008003000180020004800400050300002805010000800B0001000C00040001518080010007800E008080030001800200028004000E0300002806010000800B00 01000C00040001518080010007800E00808003000180020002800400050300002807010000800B0001000C00040001518080010007800E0100800300018002000280 04000E0000002808010000800B0001000C00040001518080010007800E01008003000180020002800400050D0000144A131C81070358455C5728F20E95452F0D0000 147D9419A65310CA6F2C179D9215529D560D000014CD60464335DF21F87CFDB2FC68B6A4480D00001490CB80913EBB696E086381B5EC427B1F0D00001416F6CA16E4 A4066D83821A0F0AEAA8620D0000144485152D18B6BBCD0BE8A8469579DDCC0D000014AFCAD71368A1F1C96B8696FC775701000D000014CA4A4CBB12EAB6C58C5706 7C2E6537860D000014A58FEC5036F57B21E8B499E336C76EE60D0000144048B7D56EBCE88525E7DE7F00D6C2D30D0000184048B7D56EBCE88525E7DE7F00D6C2D3C0 000000000000148299031757A36082C6A621DE00000000 ike V=root:0:a88b04275081853f/0000000000000000:138794: responder: main mode get 1st message... ike V=root:0:a88b04275081853f/0000000000000000:138794: VID RFC 3947 4A131C81070358455C5728F20E95452F ike V=root:0:a88b04275081853f/0000000000000000:138794: VID draft-ietf-ipsec-nat-t-ike-03 7D9419A65310CA6F2C179D9215529D56 ike V=root:0:a88b04275081853f/0000000000000000:138794: VID draft-ietf-ipsec-nat-t-ike-02 CD60464335DF21F87CFDB2FC68B6A448 ike V=root:0:a88b04275081853f/0000000000000000:138794: VID draft-ietf-ipsec-nat-t-ike-02\n 90CB80913EBB696E086381B5EC427B1F ike V=root:0:a88b04275081853f/0000000000000000:138794: VID draft-ietf-ipsec-nat-t-ike-01 16F6CA16E4A4066D83821A0F0AEAA862 ike V=root:0:a88b04275081853f/0000000000000000:138794: VID draft-ietf-ipsec-nat-t-ike-00 4485152D18B6BBCD0BE8A8469579DDCC ike V=root:0:a88b04275081853f/0000000000000000:138794: VID DPD AFCAD71368A1F1C96B8696FC77570100 ike V=root:0:a88b04275081853f/0000000000000000:138794: VID Fortinet Auto-Discovery Receiver CA4A4CBB12EAB6C58C57067C2E653786 ike V=root:0:a88b04275081853f/0000000000000000:138794: VID Fortinet Exchange Interface IP A58FEC5036F57B21E8B499E336C76EE6 ike V=root:0:a88b04275081853f/0000000000000000:138794: VID FRAGMENTATION 4048B7D56EBCE88525E7DE7F00D6C2D3 ike V=root:0:a88b04275081853f/0000000000000000:138794: VID FRAGMENTATION 4048B7D56EBCE88525E7DE7F00D6C2D3C0000000 ike V=root:0:a88b04275081853f/0000000000000000:138794: VID FORTIGATE 8299031757A36082C6A621DE00000000 ike V=root:0:a88b04275081853f/0000000000000000:138794: negotiation result ike V=root:0:a88b04275081853f/0000000000000000:138794: proposal id = 1: ike V=root:0:a88b04275081853f/0000000000000000:138794: protocol id = ISAKMP: ike V=root:0:a88b04275081853f/0000000000000000:138794: trans_id = KEY_IKE. ike V=root:0:a88b04275081853f/0000000000000000:138794: encapsulation = IKE/none ike V=root:0:a88b04275081853f/0000000000000000:138794: type=OAKLEY_ENCRYPT_ALG, val=AES_CBC, key-len=128 ike V=root:0:a88b04275081853f/0000000000000000:138794: type=OAKLEY_HASH_ALG, val=SHA2_256. ike V=root:0:a88b04275081853f/0000000000000000:138794: type=AUTH_METHOD, val=PRESHARED_KEY. ike V=root:0:a88b04275081853f/0000000000000000:138794: type=OAKLEY_GROUP, val=MODP2048. ike V=root:0:a88b04275081853f/0000000000000000:138794: ISAKMP SA lifetime=86400 ike V=root:0:a88b04275081853f/0000000000000000:138794: SA proposal chosen, matched gateway HUB-01 ike V=root:0:HUB-01:HUB-01: created connection: 0x555f6d9bd0 5 49.255.x.x->65.181.15.2:5019. ike V=root:0:HUB-01:138794: DPD negotiated ike V=root:0:HUB-01:138794: peer is FortiGate/FortiOS (v0 b0) ike V=root:0:HUB-01:138794: selected NAT-T version: RFC 3947 .... ike V=root:0:HUB-01:138794: sent IKE msg (ident_r1send): 49.255.x.x:500->65.181.15.2:5019, len=232, vrf=0, id=a88b04275081853f/0 0f6927772bceebb ike V=root:0: comes 65.181.15.2:5019->49.255.x.x:500,ifindex=5,vrf=0,len=380.... .... ike V=root:0:HUB-01:138794: responder:main mode get 2nd message... ike V=root:0:HUB-01:138794: received NAT-D payload type 20 ike V=root:0:HUB-01:138794: received NAT-D payload type 20 ike V=root:0:HUB-01:138794: NAT detected: PEER ike V=root:0:HUB-01:138794: generate DH public value request queued ike V=root:0:HUB-01:138794: compute DH shared secret request queued .... ike V=root:0:HUB-01:138794: sent IKE msg (ident_r2send): 49.255.x.x:500->65.181.15.2:5019, len=380, vrf=0, id=a88b04275081853f/0 0f6927772bceebb ike 0:HUB-01:138794: ISAKMP SA a88b04275081853f/00f6927772bceebb key 16:524919B5AE49A09B234E25CA228133CF ike V=root:0: comes 65.181.15.2:11496->49.255.x.x:4500,ifindex=5,vrf=0,len=144.... .... ike V=root:0:HUB-01:138794: responder: main mode get 3rd message... .... ike V=root:0:HUB-01:138794: received p1 notify type INITIAL-CONTACT ike V=root:0:HUB-01:138794: received p1 notify type INTERFACE-ADDR4 ike V=root:0:HUB-01:138794: INTERFACE-ADDR4 10.100.1.5 ike V=root:0:HUB-01:138794: peer identifier IPV4_ADDR 192.168.x.x ike V=root:0:HUB-01:138794: error, received '' expected 'starlink'
In most cases, the root cause is because Starlink (1) uses CG-NAT, (2) changes its IP mid-session, (3) breaks classic static IPsec assumptions. From the error provided on the IKE debug, and the HUB is expecting a peer ID match.
To resolve it: On the spoke: Configure a local ID.
config vpn ipsec phase1-interface edit Spoke-01 set localid starlink next end
On the hub: Configure Peer ID.
config vpn ipsec phase1-interface edit HUB-01 set peerid starlink set peertype one next end
This configuration change prevents FortiGate from responding to Starlink's changing IPs.
If issue is still not resolved, collect an output of mentioned debug commands, and raise a ticket with the Fortinet support team. See Customer Service Tip: How to create a ticket for Fortinet TAC.
Related articles: |