ipsec issue after firmware upgrade and apply HA at hub end

[journeyman]

FGT60C is a hub with 8 ipsec interface tunnels (all FGT60C). These were working fine but after upgrading from 4.1.4 to 4.3.18 and simultaneously applying HA, all ipsec tunnels are unstable. All remotes are already running 4.3.18. All tunnels establish and fail continuously (our syslog shows many interface was turned up|down messages). All tunnels were up immediately prior to the firmware and HA change. NAT rules etc which have previously been an issue seem OK, no lost packets (afaik) etc.

phase1-interface and phase2-interface configurations have been crosschecked OK. PSK has been reset and is correct (seen in line 45 of remote debug below).

The remote debug shows authentication success, the hub debug does not. Is this perhaps part of the issue? For that matter, the debug output seems a bit different even though both devices are running 4.3.18.

Otherwise. any suggestion for next step to diagnose?

Thanks in advance

Debug from a remote site:

001 # diag deb app ike -1
002 # diag deb en
003 ike 0: comes hub-ip:500->my-ip:500,ifindex=5....
004 ike 0: IKEv1 exchange=Identity Protection id=e24237b7f7d82b54/0000000000000000 len=244
005 ike 0: in lots-of-hex
006 ike 0:ipsec-at-remote:5207: responder: main mode get 1st message...
007 ike 0:ipsec-at-remote:5207: VID RFC 3947 4A131C81070358455C5728F20E95452F
008 ike 0:ipsec-at-remote:5207: VID draft-ietf-ipsec-nat-t-ike-03 7D9419A65310CA6F2C179D9215529D56
009 ike 0:ipsec-at-remote:5207: VID draft-ietf-ipsec-nat-t-ike-02 CD60464335DF21F87CFDB2FC68B6A448
010 ike 0:ipsec-at-remote:5207: VID draft-ietf-ipsec-nat-t-ike-02\n 90CB80913EBB696E086381B5EC427B1F
011 ike 0:ipsec-at-remote:5207: VID draft-ietf-ipsec-nat-t-ike-01 16F6CA16E4A4066D83821A0F0AEAA862
012 ike 0:ipsec-at-remote:5207: VID draft-ietf-ipsec-nat-t-ike-00 4485152D18B6BBCD0BE8A8469579DDCC
013 ike 0:ipsec-at-remote:5207: VID DPD AFCAD71368A1F1C96B8696FC77570100
014 ike 0:ipsec-at-remote:5207: DPD negotiated
015 ike 0:ipsec-at-remote:5207: VID FORTIGATE 8299031757A36082C6A621DE000402B1
016 ike 0:ipsec-at-remote:5207: peer is FortiGate/FortiOS (v4 b689)
017 ike 0:ipsec-at-remote:5207: negotiation result
018 ike 0:ipsec-at-remote:5207: proposal id = 1:
019 ike 0:ipsec-at-remote:5207: protocol id = ISAKMP:
020 ike 0:ipsec-at-remote:5207: trans_id = KEY_IKE.
021 ike 0:ipsec-at-remote:5207: encapsulation = IKE/none
022 ike 0:ipsec-at-remote:5207: type=OAKLEY_ENCRYPT_ALG, val=AES_CBC.
023 ike 0:ipsec-at-remote:5207: type=OAKLEY_HASH_ALG, val=SHA.
024 ike 0:ipsec-at-remote:5207: type=AUTH_METHOD, val=PRESHARED_KEY.
025 ike 0:ipsec-at-remote:5207: type=OAKLEY_GROUP, val=1024.
026 ike 0:ipsec-at-remote:5207: ISKAMP SA lifetime=28800
027 ike 0:ipsec-at-remote:5207: selected NAT-T version: RFC 3947
028 ike 0:ipsec-at-remote:5207: cookie e24237b7f7d82b54/05958ceab9c7cb69
029 ike 0:ipsec-at-remote:5207: out lots-of-hex
030 ike 0:ipsec-at-remote:5207: sent IKE msg (ident_r1send): my-ip:500->hub-ip:500, len=144, id=e24237b7f7d82b54/05958ceab9c7cb69
031 ike 0: comes hub-ip:500->my-ip:500,ifindex=5....
032 ike 0: IKEv1 exchange=Identity Protection id=e24237b7f7d82b54/05958ceab9c7cb69 len=228
033 ike 0: in lots-of-hex
034 ike 0:ipsec-at-remote:5207: responder:main mode get 2nd message...
035 ike 0:ipsec-at-remote:5207: NAT detected: ME 
036 ike 0:ipsec-at-remote:5207: out lots-of-hex
037 ike 0:ipsec-at-remote:5207: sent IKE msg (ident_r2send): my-ip:500->hub-ip:500, len=228, id=e24237b7f7d82b54/05958ceab9c7cb69
038 ike 0:ipsec-at-remote:5207: ISAKMP SA e24237b7f7d82b54/05958ceab9c7cb69 key 16:C9C7F81FED3D466E35FCD4A0EAB97E7A
039 ike 0: comes hub-ip:4500->my-ip:4500,ifindex=5....
040 ike 0: IKEv1 exchange=Identity Protection id=e24237b7f7d82b54/05958ceab9c7cb69 len=108
041 ike 0: in lots-of-hex
042 ike 0:ipsec-at-remote:5207: responder: main mode get 3rd message...
043 ike 0:ipsec-at-remote:5207: dec E24237B7F7D82B5405958CEAB9C7CB6905100201000000000000006C080000140200000069707365632D746F2D6368770B0000187701753AA31881780512AC6B84CD46DC21F1AAB40000001C0000000101106002E24237B7F7D82B5405958CEAB9C7CB699F89A58D863BFD07
044 ike 0:ipsec-at-remote:5207: received notify type 24578
045 ike 0:ipsec-at-remote:5207: PSK authentication succeeded
046 ike 0:ipsec-at-remote:5207: authentication OK
047 ike 0:ipsec-at-remote:5207: enc lots-of-hex
048 ike 0:ipsec-at-remote:5207: port change 500 -> 4500
049 ike 0:ipsec-at-remote:5207: out lots-of-hex
050 ike 0:ipsec-at-remote:5207: sent IKE msg (ident_r3send): my-ip:4500->hub-ip:4500, len=76, id=e24237b7f7d82b54/05958ceab9c7cb69
051 ike 0:ipsec-at-remote:5207: established IKE SA e24237b7f7d82b54/05958ceab9c7cb69
052 ike 0:ipsec-at-remote:5207: processing INITIAL-CONTACT
053 ike 0:ipsec-at-remote: flushing 
054 ike 0:ipsec-at-remote: flushed 
055 ike 0:ipsec-at-remote:5207: processed INITIAL-CONTACT
056 ike 0:ipsec-at-remote: set oper up
057 ike 0:ipsec-at-remote: schedule auto-negotiate
058 ike 0:ipsec-at-remote:5207: no pending Quick-Mode negotiations
059 ike 0:ipsec-at-remote:ipsec-at-remote-p2: IPsec SA connect 5 my-ip->hub-ip:4500
060 ike 0:ipsec-at-remote:ipsec-at-remote-p2: using existing connection
061 ike 0:ipsec-at-remote:ipsec-at-remote-p2: config found
062 ike 0:ipsec-at-remote:ipsec-at-remote-p2: IPsec SA connect 5 my-ip->hub-ip:4500 negotiating
063 ike 0:ipsec-at-remote: carrier up
064 ike 0:ipsec-at-remote:5207: cookie e24237b7f7d82b54/05958ceab9c7cb69:6eab0c3a
065 ike 0:ipsec-at-remote:5207:ipsec-at-remote-p2:628914: natt flags 0x23, encmode 1->3
066 ike 0:ipsec-at-remote:5207:ipsec-at-remote-p2:628914: initiator selectors 0 0:0.0.0.0/0.0.0.0:0:0->0:0.0.0.0/0.0.0.0:0:0
067 ike 0:ipsec-at-remote:5207: enc lots-of-hex
068 ike 0:ipsec-at-remote:5207: out lots-of-hex
069 ike 0:ipsec-at-remote:5207: sent IKE msg (quick_i1send): my-ip:4500->hub-ip:4500, len=300, id=e24237b7f7d82b54/05958ceab9c7cb69:6eab0c3a
070 ike 0:ipsec-at-remote:5207: out lots-of-hex
071 ike 0:ipsec-at-remote:5207: sent IKE msg (P2_RETRANSMIT): my-ip:4500->hub-ip:4500, len=300, id=e24237b7f7d82b54/05958ceab9c7cb69:6eab0c3a
072 ike 0:ipsec-at-remote: link is idle 5 my-ip->hub-ip:4500 dpd=1 seqno=530cf
073 ike 0:ipsec-at-remote:5207: send IKEv1 DPD probe, seqno 340175
074 ike 0:ipsec-at-remote:5207: enc lots-of-hex
075 ike 0:ipsec-at-remote:5207: out lots-of-hex
076 ike 0:ipsec-at-remote:5207: sent IKE msg (R-U-THERE): my-ip:4500->hub-ip:4500, len=92, id=e24237b7f7d82b54/05958ceab9c7cb69:da5ac04c
077 ike 0:ipsec-at-remote:5207: out lots-of-hex
078 ike 0:ipsec-at-remote:5207: sent IKE msg (P2_RETRANSMIT): my-ip:4500->hub-ip:4500, len=300, id=e24237b7f7d82b54/05958ceab9c7cb69:6eab0c3a
079 ike 0:ipsec-at-remote:ipsec-at-remote-p2: IPsec SA connect 5 my-ip->hub-ip:4500
080 ike 0:ipsec-at-remote:ipsec-at-remote-p2: using existing connection
081 ike 0:ipsec-at-remote:ipsec-at-remote-p2: config found
082 ike 0: comes hub-ip:4500->my-ip:4500,ifindex=5....
083 ike 0: IKEv1 exchange=Identity Protection id=e24237b7f7d82b54/05958ceab9c7cb69 len=108
084 ike 0: in lots-of-hex
085 ike 0:ipsec-at-remote:5207: retransmission, re-send last message
086 ike 0:ipsec-at-remote:5207: out lots-of-hex
087 ike 0:ipsec-at-remote:5207: sent IKE msg (retransmit): my-ip:4500->hub-ip:4500, len=76, id=e24237b7f7d82b54/05958ceab9c7cb69
088 ike 0:ipsec-at-remote:ipsec-at-remote-p2: IPsec SA connect 5 my-ip->hub-ip:4500
089 ike 0:ipsec-at-remote:ipsec-at-remote-p2: using existing connection
090 ike 0:ipsec-at-remote:ipsec-at-remote-p2: config found
091 ike 0:ipsec-at-remote: link is idle 5 my-ip->hub-ip:4500 dpd=1 seqno=530cf
092 ike 0:ipsec-at-remote:5207: send IKEv1 DPD probe, seqno 340175
093 ike 0:ipsec-at-remote:5207: enc lots-of-hex
094 ike 0:ipsec-at-remote:5207: out lots-of-hex
095 ike 0:ipsec-at-remote:5207: sent IKE msg (R-U-THERE): my-ip:4500->hub-ip:4500, len=92, id=e24237b7f7d82b54/05958ceab9c7cb69:27ffdb1b
096 ike 0:ipsec-at-remote:ipsec-at-remote-p2: IPsec SA connect 5 my-ip->hub-ip:4500
097 ike 0:ipsec-at-remote:ipsec-at-remote-p2: using existing connection
098 ike 0:ipsec-at-remote:ipsec-at-remote-p2: config found
099 ike 0:ipsec-at-remote:5207: out lots-of-hex
100 ike 0:ipsec-at-remote:5207: sent IKE msg (P2_RETRANSMIT): my-ip:4500->hub-ip:4500, len=300, id=e24237b7f7d82b54/05958ceab9c7cb69:6eab0c3a
101 ike 0:ipsec-at-remote: link is idle 5 my-ip->hub-ip:4500 dpd=1 seqno=530cf
102 ike 0:ipsec-at-remote:5207: send IKEv1 DPD probe, seqno 340175
103 ike 0:ipsec-at-remote:5207: enc lots-of-hex
104 ike 0:ipsec-at-remote:5207: out lots-of-hex
105 ike 0:ipsec-at-remote:5207: sent IKE msg (R-U-THERE): my-ip:4500->hub-ip:4500, len=92, id=e24237b7f7d82b54/05958ceab9c7cb69:afd2468c
106 ike 0: comes hub-ip:4500->my-ip:4500,ifindex=5....
107 ike 0: IKEv1 exchange=Identity Protection id=e24237b7f7d82b54/05958ceab9c7cb69 len=108
108 ike 0: in lots-of-hex
109 ike 0:ipsec-at-remote:5207: retransmission, re-send last message
110 ike 0:ipsec-at-remote:5207: out lots-of-hex
111 ike 0:ipsec-at-remote:5207: sent IKE msg (retransmit): my-ip:4500->hub-ip:4500, len=76, id=e24237b7f7d82b54/05958ceab9c7cb69
112 ike 0:ipsec-at-remote:ipsec-at-remote-p2: IPsec SA connect 5 my-ip->hub-ip:4500
113 ike 0:ipsec-at-remote:ipsec-at-remote-p2: using existing connection
114 ike 0:ipsec-at-remote:ipsec-at-remote-p2: config found
115 ike 0:ipsec-at-remote: carrier down
116 ike 0:ipsec-at-remote: set oper down
117 ike 0:ipsec-at-remote: deleting
118 ike 0:ipsec-at-remote: flushing 
119 ike 0:ipsec-at-remote: flushed 
120 ike 0:ipsec-at-remote:5207: send ISAKMP delete e24237b7f7d82b54/05958ceab9c7cb69
121 ike 0:ipsec-at-remote:5207: enc lots-of-hex
122 ike 0:ipsec-at-remote:5207: out lots-of-hex
123 ike 0:ipsec-at-remote:5207: sent IKE msg (ISKAMP SA DELETE-NOTIFY): my-ip:4500->hub-ip:4500, len=92, id=e24237b7f7d82b54/05958ceab9c7cb69:94c120a8
124 ike 0:ipsec-at-remote: reset NAT-T
125 ike 0:ipsec-at-remote: deleted
126 ike 0:ipsec-at-remote: schedule auto-negotiate
127 ike 0:ipsec-at-remote: link fail 5 my-ip->hub-ip:4500 dpd=1
128 ike 0:ipsec-at-remote: ignoring since port is not 500
129 ike 0:ipsec-at-remote: reset NAT-T settings
130 ike shrank heap by 131072 bytes
131 ike 0:ipsec-at-remote: auto-negotiate connection
132 ike 0:ipsec-at-remote: created connection: 0x1d2a260 5 my-ip->hub-ip:500.
133 ike 0:ipsec-at-remote:5208: initiator: main mode is sending 1st message...
134 ike 0:ipsec-at-remote:5208: cookie 20a7e7d08c993331/0000000000000000
135 ike 0:ipsec-at-remote:5208: out lots-of-hex
136 ike 0:ipsec-at-remote:5208: sent IKE msg (ident_i1send): my-ip:500->hub-ip:500, len=244, id=20a7e7d08c993331/0000000000000000
137 ike 0:ipsec-at-remote:5208: out lots-of-hex
138 ike 0:ipsec-at-remote:5208: sent IKE msg (P1_RETRANSMIT): my-ip:500->hub-ip:500, len=244, id=20a7e7d08c993331/0000000000000000
139 ike 0: comes hub-ip:500->my-ip:500,ifindex=5....
140 ike 0: IKEv1 exchange=Identity Protection id=ada75449e872ca9a/0000000000000000 len=244
141 ike 0: in lots-of-hex
142 ike 0: found ipsec-at-remote my-ip 5 -> hub-ip:500
143 ike 0:ipsec-at-remote:5209: responder: main mode get 1st message...

Debug from the hub site - note, no mention of psk success

001 # diag vpn ike log-filter list 
002 vd: any
003 name: ipsec-at-hub
004 interface: any
005 IPv4 source: any
006 IPv4 dest: rem-nat-ip
007 IPv6 source: any
008 IPv6 dest: any
009 source port: any
010 dest port: any
011 
012 # diag deb app ike -1
013 # diag deb en
014 
015 
016 ike 0: comes rem-nat-ip:500->hub-ip:500,ifindex=8....
017 ike 0: IKEv1 exchange=Identity Protection id=0e28555e25dd2dd9/ab80a5ace2a1c17e len=144
018 ike 0: in lots-of-hex
019 ike 0:ipsec-at-hub:9054: initiator: main mode get 1st response...
020 ike 0:ipsec-at-hub:9054: VID RFC 3947 4A131C81070358455C5728F20E95452F
021 ike 0:ipsec-at-hub:9054: VID DPD AFCAD71368A1F1C96B8696FC77570100
022 ike 0:ipsec-at-hub:9054: DPD negotiated
023 ike 0:ipsec-at-hub:9054: VID FORTIGATE 8299031757A36082C6A621DE000402B1
024 ike 0:ipsec-at-hub:9054: peer is FortiGate/FortiOS (v4 b689)
025 ike 0:ipsec-at-hub:9054: selected NAT-T version: RFC 3947
026 ike 0:ipsec-at-hub:9054: negotiation result
027 ike 0:ipsec-at-hub:9054: proposal id = 1:
028 ike 0:ipsec-at-hub:9054:   protocol id = ISAKMP:
029 ike 0:ipsec-at-hub:9054:      trans_id = KEY_IKE.
030 ike 0:ipsec-at-hub:9054:      encapsulation = IKE/none
031 ike 0:ipsec-at-hub:9054:         type=OAKLEY_ENCRYPT_ALG, val=AES_CBC.
032 ike 0:ipsec-at-hub:9054:         type=OAKLEY_HASH_ALG, val=SHA.
033 ike 0:ipsec-at-hub:9054:         type=AUTH_METHOD, val=PRESHARED_KEY.
034 ike 0:ipsec-at-hub:9054:         type=OAKLEY_GROUP, val=1024.
035 ike 0:ipsec-at-hub:9054: ISKAMP SA lifetime=28800
036 ike 0:ipsec-at-hub:9054: out lots-of-hex
037 ike 0:ipsec-at-hub:9054: sent IKE msg (ident_i2send): hub-ip:500->rem-nat-ip:500, len=228, id=0e28555e25dd2dd9/ab80a5ace2a1c17e
038 ike 0: comes rem-nat-ip:500->hub-ip:500,ifindex=8....
039 ike 0: IKEv1 exchange=Identity Protection id=0e28555e25dd2dd9/ab80a5ace2a1c17e len=144
040 ike 0: in lots-of-hex
041 ike 0:ipsec-at-hub:9054: retransmission, re-send last message
042 ike 0:ipsec-at-hub:9054: out lots-of-hex
043 ike 0:ipsec-at-hub:9054: sent IKE msg (retransmit): hub-ip:500->rem-nat-ip:500, len=228, id=0e28555e25dd2dd9/ab80a5ace2a1c17e
044 ike 0: comes rem-nat-ip:500->hub-ip:500,ifindex=8....
045 ike 0: IKEv1 exchange=Identity Protection id=0e28555e25dd2dd9/ab80a5ace2a1c17e len=228
046 ike 0: in lots-of-hex
047 ike 0:ipsec-at-hub:9054: initiator: main mode get 2nd response...
048 ike 0:ipsec-at-hub:9054: NAT detected: PEER
049 ike 0:ipsec-at-hub:9054: NAT-T float port 4500
050 ike 0:ipsec-at-hub:9054: ISAKMP SA 0e28555e25dd2dd9/ab80a5ace2a1c17e key 16:D002ADFCCECD76D00CADE8B465395119
051 ike 0:ipsec-at-hub:9054: add INITIAL-CONTACT
052 ike 0:ipsec-at-hub:9054: enc lots-of-hex
053 ike 0:ipsec-at-hub:9054: out lots-of-hex
054 ike 0:ipsec-at-hub:9054: sent IKE msg (ident_i3send): hub-ip:4500->rem-nat-ip:4500, len=108, id=0e28555e25dd2dd9/ab80a5ace2a1c17e
055 ike 0: comes rem-nat-ip:500->hub-ip:500,ifindex=8....
056 ike 0: IKEv1 exchange=Identity Protection id=0e28555e25dd2dd9/ab80a5ace2a1c17e len=228
057 ike 0: in lots-of-hex
058 ike 0:ipsec-at-hub:9054: retransmission, re-send last message
059 ike 0:ipsec-at-hub:9054: out lots-of-hex
060 ike 0:ipsec-at-hub:9054: sent IKE msg (retransmit): hub-ip:4500->rem-nat-ip:4500, len=108, id=0e28555e25dd2dd9/ab80a5ace2a1c17e
061 ike 0:ipsec-at-hub:9054: out lots-of-hex
062 ike 0:ipsec-at-hub:9054: sent IKE msg (P1_RETRANSMIT): hub-ip:4500->rem-nat-ip:4500, len=108, id=0e28555e25dd2dd9/ab80a5ace2a1c17e
063 ike shrank heap by 122880 bytes
064 ike 0:ipsec-at-hub: NAT keep-alive 8 hub-ip->rem-nat-ip:4500.
065 ike 0:ipsec-at-hub:9054: negotiation timeout, deleting
066 ike 0:ipsec-at-hub: connection expiring due to phase1 down
067 ike 0:ipsec-at-hub: deleting
068 ike 0:ipsec-at-hub: flushing 
069 ike 0:ipsec-at-hub: flushed 
070 ike 0:ipsec-at-hub: reset NAT-T
071 ike 0:ipsec-at-hub: deleted
072 ike 0:ipsec-at-hub: schedule auto-negotiate
073 ike 0:ipsec-at-hub:9069: initiator: main mode is sending 1st message...
074 ike 0:ipsec-at-hub:9069: cookie da08c6777fc5160d/0000000000000000
075 ike 0:ipsec-at-hub:9069: out lots-of-hex
076 ike 0:ipsec-at-hub:9069: sent IKE msg (ident_i1send): hub-ip:500->rem-nat-ip:500, len=244, id=da08c6777fc5160d/0000000000000000
077 ike 0: comes rem-nat-ip:4500->hub-ip:500,ifindex=8....
078 ike 0:ipsec-at-hub:9069: out lots-of-hex
079 ike 0:ipsec-at-hub:9069: sent IKE msg (P1_RETRANSMIT): hub-ip:500->rem-nat-ip:500, len=244, id=da08c6777fc5160d/0000000000000000
080 ike 0: comes rem-nat-ip:4500->hub-ip:500,ifindex=8....
081 ike 0: comes rem-nat-ip:4500->hub-ip:500,ifindex=8....
082 ike 0:ipsec-at-hub:9069: out lots-of-hex
083 ike 0:ipsec-at-hub:9069: sent IKE msg (P1_RETRANSMIT): hub-ip:500->rem-nat-ip:500, len=244, id=da08c6777fc5160d/0000000000000000
084 ike 0:ipsec-at-hub:9069: negotiation timeout, deleting
085 ike 0:ipsec-at-hub: connection expiring due to phase1 down
086 ike 0:ipsec-at-hub: deleting
087 ike 0:ipsec-at-hub: flushing 
088 ike 0:ipsec-at-hub: flushed 
089 ike 0:ipsec-at-hub: deleted
090 ike 0:ipsec-at-hub: schedule auto-negotiate
091 ike 0:ipsec-at-hub:9085: initiator: main mode is sending 1st message...
092 ike 0:ipsec-at-hub:9085: cookie f7b7009c1f709f12/0000000000000000
093 ike 0:ipsec-at-hub:9085: out lots-of-hex
094 ike 0:ipsec-at-hub:9085: sent IKE msg (ident_i1send): hub-ip:500->rem-nat-ip:500, len=244, id=f7b7009c1f709f12/0000000000000000
095 ike 0:ipsec-at-hub:9085: out lots-of-hex
096 ike 0:ipsec-at-hub:9085: sent IKE msg (P1_RETRANSMIT): hub-ip:500->rem-nat-ip:500, len=244, id=f7b7009c1f709f12/0000000000000000
097 ike 0:ipsec-at-hub:9085: out lots-of-hex
098 ike 0:ipsec-at-hub:9085: sent IKE msg (P1_RETRANSMIT): hub-ip:500->rem-nat-ip:500, len=244, id=f7b7009c1f709f12/0000000000000000

Any suggestions

[emnoc]

Op,

diag vpn tunnel list name < insert name hub> and the spoke in question

show vpn ipsec phase1-interface

show vpn ipsec phase2-interface

( for both hub and spoke )

show router static ( hub & spoke )

Also since you have HA,  disable the stand-by or wack to the stand-by t o ensure if either one is the cause of the issue.

[ede_pfau]

hi Martin,

as you've already scrutinized the configs closely I re-read your original post. Trouble only began after forming the HA cluster, right? I think I would risk having no HA for a while and switch off the slave to see if this is cluster related or not.

If you want to keep one unit as primary, even after powering up the slave at some later time, then just set the HA override on the primary unit.

It might even make sense to switch units i.e. to test with the primary alone, rebuild the cluster, and switch off the primary and re-test.

All this only makes sense if the correctness of the VPN setup is believed to be > 99%.

There are just too many factors in this equation. Even the routers involved might be the culprits, not so much the WAN router but the 3G thing. The change from phase1 to phase2 interacts with the router's NAT handling and you never know how well the firmware is designed.

But of course this is the most difficult thing to test as you cannot just swap the hardware for something similar.

[journeyman]

The problem occurred after we powered down the old unit and swapped in a replacement. The replacement had already been carefully upgraded to 4.3.18 and then configured for HA. When the replacement was powered on (HA master but no slave present) the tunnels did not come up even at this stage.

We intend to remove the HA configuration once we can get someone on site to remove power after an exec shut.

Diag vpn tunnel list output is posted above under the heading VPN Diagnostics.

Phase2 auto-negotiate default was disable, has been set to enable today with no effect.

Hub relevant full configuration showing one spoke is as follows:

config vpn ipsec phase1-interface
    edit "ipsec-at-hub"
        set type static
        set interface "internal4"
        set ip-version 4
        set ike-version 1
        set local-gw 0.0.0.0
        set nattraversal enable
        set dhgrp 2
        set keylife 28800
        set authmethod psk
        set peertype any
        set xauthtype disable
        set mode main
        set mode-cfg disable
        set proposal aes128-sha1
        set localid ''
        set localid-type auto
        set negotiate-timeout 30
        set dpd enable
        set forticlient-enforcement disable
        set remote-gw rem-nat-ip
        set monitor-phase1 ''
        set add-gw-route disable
        set psksecret ENC <112 bytes encoded psk>
        set keepalive 10
        set auto-negotiate enable
        set dpd-retrycount 3
        set dpd-retryinterval 5
    next
end
config vpn ipsec phase2-interface
    edit "ipsec-at-hub-p2"
        set auto-negotiate enable
        set dst-addr-type subnet
        set dst-port 0
        set encapsulation tunnel-mode
        set keepalive disable
        set keylife-type seconds
        set pfs enable
        set phase1name "ipsec-at-hub"
        set proposal aes128-sha1
        set protocol 0
        set replay enable
        set src-addr-type subnet
        set src-port 0
        set dhgrp 2
        set dst-subnet 0.0.0.0 0.0.0.0
        set keylifeseconds 1800
        set src-subnet 0.0.0.0 0.0.0.0
    next
end
config router static
    edit 9
        set device "internal4"
        set dst rem-nat-ip 255.255.255.255
        set gateway hub-ip-gw
        set weight 50
    next
end

Typical spoke full configuration:

config vpn ipsec phase1-interface
    edit "ipsec-at-remote"
        set type static
        set interface "wan2"
        set ip-version 4
        set ike-version 1
        set local-gw 0.0.0.0
        set nattraversal enable
        set dhgrp 2
        set keylife 28800
        set authmethod psk
        set peertype any
        set xauthtype disable
        set mode main
        set mode-cfg disable
        set proposal aes128-sha1
        set localid ''
        set localid-type auto
        set negotiate-timeout 30
        set dpd enable
        set forticlient-enforcement disable
        set remote-gw hub-ip
        set monitor-phase1 ''
        set add-gw-route disable
        set psksecret ENC <112 bytes encoded psk>
        set keepalive 10
        set auto-negotiate enable
        set dpd-retrycount 3
        set dpd-retryinterval 5
    next
end
config vpn ipsec phase2-interface
    edit "ipsec-at-remote-p2"
        set auto-negotiate enable
        set dst-addr-type subnet
        set dst-port 0
        set encapsulation tunnel-mode
        set keepalive disable
        set keylife-type seconds
        set pfs enable
        set phase1name "ipsec-at-remote"
        set proposal aes128-sha1
        set protocol 0
        set replay enable
        set src-addr-type subnet
        set src-port 0
        set dhgrp 2
        set dst-subnet 0.0.0.0 0.0.0.0
        set keylifeseconds 1800
        set src-subnet 0.0.0.0 0.0.0.0
    next
end
config router static
    edit 1
        set device "wan2"
        set dst hub-ip 255.255.255.255
        set gateway my-ip-gw
    next
end

Trouble shooting:

Configurations match

Peers reachable

Checked router ACLs etc are OK in traversed routers

No packets missing (diag sni pac <if> 'host <other-ip>' 6 0 a both ends | fgt2eth | mergecap | Tools > Compare...)

psk auth success

nat-t negotiated

phase2 dst subnet and src subnet are not set as each end has a lot of subnets to handle (hence ospf)

What can cause phase2 to not come up?

[ede_pfau]

2 remarks:

1. phase2, set keepalive enable

no reason why this should not be enabled

2. your static routes look strange (host routes?). I had expected

on the hub:

[ul]

one static route for default gateway, gateway=ISP router, IF=wanX

one static route per spoke for the private network behind spoke, no gateway, just the tunnel interface[/ul]

on a spoke:

[ul]

one static route for default gateway, gateway=ISP router, IF=wanX

one static route for the private network behind hub, no gateway, just the tunnel interface

more of these if you want to reach other spoke's networks[/ul]

I know that with n spokes you end up with n² routes if you want a full mesh, and that is probably the reason for OSPF. But at least the default routes need to be in place to reach the VPN gateway. As I don't have the meaning of the placeholders I cannot fully check the static routes given in your config.

[emnoc]

Okay

set  local-gateway 0.0.0.0 bothers me.  I never seen that in real-life, I guess it would work but for a static defined vpn it should be each peer.

Also I don't see any set remote-gateway defined  ( maybe you sanitized this  or my eyes are bad )

As ede point out, that static route is incorrect it should be ;

config router static
    edit 1
        set device "YOUR PHASE1 INTERFACE NAME "
        set dst hub-ip 255.255.255.255
        set gateway my-ip-gw   < you don't need this nor do I know what this is for 
    next
end

Also you would be ripe to use OSPF between hub and spokes for routing imho ( optional )

Ken

[journeyman]

The static routes are OK, I am sure. All traffic is amongst our own sites / hardware / networks and we don't "do" default [strike]gateways[/strike] oops, routes [edit]. The telco network is "private" and we cannot see the internet.

At the hub, internal4 connects to a LAN which has faces a frame relay router onto the relevant wan. The frame relay router knows the upsteam routes to the remote sites, so the correct route is <remote-ip/32> via <wan-router-inside> (aka <hub-ip-gw> in the configs above) on internal4. The LAN subnet is advertised on the WAN cloud.

The use of internal4 is due to wan1 and wan2 being already in use on other wan type networks.

At the remote sites, wan1 is the primary uplink (usually microwave radio), wan2 is a /30 to the 3G router for backup. To get nice failover between wan1 and wan2 we want ospf via 3G hence the tunnel. The whole design is based on an ospf failover solution in the cookbook and was working perfectly. The 3G is the nat device and can route to the hub LAN, so at the remote FGT the route is <hub-ip/32> via <3G-router-inside> (aka my-ip-gw in the configs above) on wan2.

Once the tunnel is established ospf takes over to handle the rest. In general we don't need to go spoke-to-spoke and have not created a mesh.

It was all working with the hub at firmware 4.1.4.

Thanks for the ph2 keepalive comment, happy to change. As mentioned, I've not looked closely at ph2 until now.

local-gw bothered me too. But I read the cli again today and it states that if unset it uses the ip from the interface settings. The description implies it is intended for picking up / forcing use of a secondary-ip. That's not relevant at either end so it has been left blank. Ditto happy to change. I almost did but it didn't seem worthwhile given the cli doco.

Specifying local-gw solved a recent issue involving a secondary-ip (again following upgrade to 4.3.18) so perhaps it really does need to be specified. Will try tomorrow.

Remote-gw is in there :)

Any points I've not replied to please let me know.

And thanks again.

[emnoc]

I never set the local-gw but it should be used if you have multiple address like secondaries. Why don't you set it from 0.0.0.0 any and defined it as the actual address you want the peers to use. Do this in each spoke and hub and this should match what the peers are actually  set in the "  set remote-gw rem-nat-ip" entry.

On the gateway, I still don't see how traffic of interest is going to route out a ipsec-tunnel.

e.g ( using our sanitized cfg )

    edit 1
        set device "wan2"
        set dst hub-ip 255.255.255.255
        set gateway my-ip-gw
    next
end

The above would imply that traffic for network "dst-hub-ip /32" is to be carried out of wan2 using next-hop of my-ip-gateway whatever that gateway is. This would not route thru a vpn-tunnel.

You can double check this by dumping the RIB

e.g

get router  info routing-table static

S*      0.0.0.0/0 [10/0] via x.x.x.x, wan1 S       10.0.0.0/8 [1/0] is a summary, Null S       10.0.0.1/32 [10/0] is directly connected, ssl.root S       10.0.0.2/32 [10/0] is directly connected, ssl.root S       10.0.0.3/32 [10/0] is directly connected, ssl.root S       10.0.0.4/32 [10/0] is directly connected, ssl.root S       10.0.0.5/32 [10/0] is directly connected, ssl.root S       10.0.0.6/32 [10/0] is directly connected, ssl.root S       10.0.0.7/32 [10/0] is directly connected, ssl.root S       10.0.0.8/32 [10/0] is directly connected, ssl.root S       10.0.0.9/32 [10/0] is directly connected, ssl.root S       10.0.0.10/32 [10/0] is directly connected, ssl.root S       10.20.1.67/32 [10/0] is directly connected, tunnelciscoASA01 S       169.254.0.0/16 [254/0] is a summary, Null S       172.16.0.0/12 [254/0] is a summary, Null S       172.16.1.0/24 [10/0] is directly connected, tunnelH01 S       172.16.2.0/24 [10/0] is directly connected, tunnelJ01 S       192.0.0.0/24 [254/0] is a summary, Null S       192.0.2.0/24 [254/0] is a summary, Null S       198.18.0.0/15 [254/0] is a summary, Null S       198.51.100.0/24 [254/0] is a summary, Null S       203.0.113.0/24 [254/0] is a summary, Null

The routes to the vpn are to devices  "tunnelXXXX"

I would still like to see a diag vpn tunnel list 

e.g

diag vpn tunnel list name  < insert your name here phase1>

And a diag debug of traffic you expected to go over a tunnel while doing a ping

diag debug dis

diag debug reset

diag debug flow filter addr x.x.x.x

diag debg flow show console enable

diag debug flow trace start 100

diag debug en

Diag debug flow is really  your best friend ;)

[ede_pfau]

The 'local-gw' setting is correct and usually is never set explicitly unless you want to terminate the VPN on a secondary IP address of your WAN port. I'd leave it as it is.

Routing...to the public IPs must be correct as the FGTs connect but routing between the protected subnets is obscure. You have more insight in these details - just make sure there is a route to the remote private LAN behind the tunnel, on both sides, or else the FGT will discard those packets as being 'from an unknown source' (RPF).

Before running out of clues you could do a debug session with 'debug flow'. You would see how packets with destination hub are handled on a spoke which covers Quick Mode selectors and routing in one step.

[emnoc]

Ede

Your 100% correct the show full output has my set as 0.0.0.0 also. I guess I never noticed it

The routing is surely an issues and needs to be address b4 he continues. I believe the ike/ipsec proposal/psk would not be an issues as of this stage.

[journeyman]

phase2-interface keepalive set to enable, no success.

phase1-interface local-gw left at default.

HA configuration removed, no success. Now running a single FGT, the other has no power. BTW HA was removed simply using set mode standalone with no other changes.

The output for diag vpn tunnel is given above, repeated for convenience:

Hub end -

# diag vpn ike gateway list name ipsec-at-hub
vd: root/0
name: ipsec-at-hub
version: 1
interface: internal4 8
addr: hub-ip:500 -> rem-nat-ip:500
created: 29s ago
IKE SA: created 1/1
IPsec SA: created 0/0

  id/spi: 125946 c99da1215d72bdef/0000000000000000
  direction: responder
  status: connecting, state 3, started 29s ago

# diag vpn tunnel list name ipsec-at-hub
list ipsec tunnel by names in vd 0
------------------------------------------------------
name=ipsec-at-hub ver=1 serial=3 hub-ip:0->rem-nat-ip:0 lgwy=dyn tun=intf mode=auto bound_if=8
proxyid_num=1 child_num=0 refcnt=6 ilast=9 olast=9
stat: rxp=0 txp=0 rxb=0 txb=0
dpd: mode=active on=0 idle=5000ms retry=3 count=0 seqno=0
natt: mode=none draft=0 interval=0 remote_port=0
proxyid=ipsec-at-hub-p2 proto=0 sa=0 ref=1 auto_negotiate=0 serial=1 
  src: 0:0.0.0.0/0.0.0.0:0
  dst: 0:0.0.0.0/0.0.0.0:0
 
 
 
# diag vpn ike gateway list name ipsec-at-hub
vd: root/0
name: ipsec-at-hub
version: 1
interface: internal4 8
addr: hub-ip:4500 -> rem-nat-ip:500
created: 23s ago
IKE SA: created 1/1
IPsec SA: created 0/0

  id/spi: 125960 02dab0a9af15ac14/8b8cd3ea381f9a82
  direction: responder
  status: connecting, state 7, started 23s ago

# diag vpn tunnel list name ipsec-at-hub
list ipsec tunnel by names in vd 0
------------------------------------------------------
name=ipsec-at-hub ver=1 serial=3 hub-ip:4500->rem-nat-ip:4500 lgwy=dyn tun=intf mode=auto bound_if=8
proxyid_num=1 child_num=0 refcnt=7 ilast=28 olast=8
stat: rxp=0 txp=0 rxb=0 txb=0
dpd: mode=active on=0 idle=5000ms retry=3 count=0 seqno=0
natt: mode=keepalive draft=32 interval=10 remote_port=4500
proxyid=ipsec-at-hub-p2 proto=0 sa=0 ref=1 auto_negotiate=0 serial=1 
  src: 0:0.0.0.0/0.0.0.0:0
  dst: 0:0.0.0.0/0.0.0.0:0

Remote -

# diag vpn ike gateway list name ipsec-at-remote
vd: root/0
name: ipsec-at-remote
version: 1
interface: wan2 5
addr: my-ip:500 -> hub-ip:500
created: 1s ago
IKE SA: created 2/2
IPsec SA: created 0/0

  id/spi: 20678 02dab0a9af15ac14/8b8cd3ea381f9a82
  direction: responder
  status: connecting, state 3, started 1s ago

  id/spi: 20677 2d6806218aa4643b/0000000000000000
  direction: responder
  status: connecting, state 3, started 1s ago

# diag vpn tunnel list name ipsec-at-remote
list all ipsec tunnel in vd 0
------------------------------------------------------
name=ipsec-at-remote ver=1 serial=1 my-ip:0->hub-ip:0 lgwy=dyn tun=intf mode=auto bound_if=5
proxyid_num=1 child_num=0 refcnt=8 ilast=4 olast=4
stat: rxp=0 txp=0 rxb=0 txb=0
dpd: mode=active on=0 idle=5000ms retry=3 count=0 seqno=343794
natt: mode=none draft=0 interval=0 remote_port=0
proxyid=ipsec-at-remote-p2 proto=0 sa=0 ref=1 auto_negotiate=0 serial=1 
  src: 0:0.0.0.0/0.0.0.0:0
  dst: 0:0.0.0.0/0.0.0.0:0

# diag vpn ike gateway list name ipsec-at-remote
vd: root/0
name: ipsec-at-remote
version: 1
interface: wan2 5
addr: my-ip:4500 -> hub-ip:4500
created: 30s ago
IKE SA: created 1/2  established 1/1  time 19010/19010/19010 ms
IPsec SA: created 1/1

  id/spi: 20678 02dab0a9af15ac14/8b8cd3ea381f9a82
  direction: responder
  status: established 29-10s ago = 19010ms
  proposal: aes128-sha1
  key: 683dcc65337afe6f-cbb772567361d43e
  lifetime/rekey: 28800/28519
  DPD sent/recv: 00053ef3/00000000

# diag vpn tunnel list name ipsec-at-remote
list all ipsec tunnel in vd 0
------------------------------------------------------
name=ipsec-at-remote ver=1 serial=1 my-ip:4500->hub-ip:4500 lgwy=dyn tun=intf mode=auto bound_if=5
proxyid_num=1 child_num=0 refcnt=10 ilast=2 olast=3
stat: rxp=0 txp=0 rxb=0 txb=0
dpd: mode=active on=1 idle=5000ms retry=3 count=2 seqno=343795
natt: mode=keepalive draft=32 interval=10 remote_port=4500
proxyid=ipsec-at-remote-p2 proto=0 sa=0 ref=1 auto_negotiate=0 serial=1 
  src: 0:0.0.0.0/0.0.0.0:0
  dst: 0:0.0.0.0/0.0.0.0:0

I don't follow the concern with routing with respect to establishing the tunnels. But to clarify (see attached diagram)...

FGT1 has a static route to A.234 via X.2 on interface internal4.

The 3G router at A.234 does the NAT and port forwards to FGT2 at Y.1

FGT2 has a static route to X.1 via Y.2 in interface wan2

Once the tunnel is up, ospf establishes (with a higher cost that the primary link). Under normal circumstances there is no traffic on the backup link other than ospf traffic, everything is via the primary link. So we can't ping down the tunnel (other than each FGT can ping the tunnel interface IP - well, if the tunnel was up).

Does that satisfy the routing concerns?

I have not had a chance to do diag debug flow today (I guess set the filter to suit ospf traffic for the tunnel).

Is there something I've missed in the vpn trouble shooting?

If phase2 isn't establishing, what could cause that?