ipsec issue after firmware upgrade and apply HA at hub end

[journeyman]

FGT60C is a hub with 8 ipsec interface tunnels (all FGT60C). These were working fine but after upgrading from 4.1.4 to 4.3.18 and simultaneously applying HA, all ipsec tunnels are unstable. All remotes are already running 4.3.18. All tunnels establish and fail continuously (our syslog shows many interface was turned up|down messages). All tunnels were up immediately prior to the firmware and HA change. NAT rules etc which have previously been an issue seem OK, no lost packets (afaik) etc.

phase1-interface and phase2-interface configurations have been crosschecked OK. PSK has been reset and is correct (seen in line 45 of remote debug below).

The remote debug shows authentication success, the hub debug does not. Is this perhaps part of the issue? For that matter, the debug output seems a bit different even though both devices are running 4.3.18.

Otherwise. any suggestion for next step to diagnose?

Thanks in advance

Debug from a remote site:

001 # diag deb app ike -1
002 # diag deb en
003 ike 0: comes hub-ip:500->my-ip:500,ifindex=5....
004 ike 0: IKEv1 exchange=Identity Protection id=e24237b7f7d82b54/0000000000000000 len=244
005 ike 0: in lots-of-hex
006 ike 0:ipsec-at-remote:5207: responder: main mode get 1st message...
007 ike 0:ipsec-at-remote:5207: VID RFC 3947 4A131C81070358455C5728F20E95452F
008 ike 0:ipsec-at-remote:5207: VID draft-ietf-ipsec-nat-t-ike-03 7D9419A65310CA6F2C179D9215529D56
009 ike 0:ipsec-at-remote:5207: VID draft-ietf-ipsec-nat-t-ike-02 CD60464335DF21F87CFDB2FC68B6A448
010 ike 0:ipsec-at-remote:5207: VID draft-ietf-ipsec-nat-t-ike-02\n 90CB80913EBB696E086381B5EC427B1F
011 ike 0:ipsec-at-remote:5207: VID draft-ietf-ipsec-nat-t-ike-01 16F6CA16E4A4066D83821A0F0AEAA862
012 ike 0:ipsec-at-remote:5207: VID draft-ietf-ipsec-nat-t-ike-00 4485152D18B6BBCD0BE8A8469579DDCC
013 ike 0:ipsec-at-remote:5207: VID DPD AFCAD71368A1F1C96B8696FC77570100
014 ike 0:ipsec-at-remote:5207: DPD negotiated
015 ike 0:ipsec-at-remote:5207: VID FORTIGATE 8299031757A36082C6A621DE000402B1
016 ike 0:ipsec-at-remote:5207: peer is FortiGate/FortiOS (v4 b689)
017 ike 0:ipsec-at-remote:5207: negotiation result
018 ike 0:ipsec-at-remote:5207: proposal id = 1:
019 ike 0:ipsec-at-remote:5207: protocol id = ISAKMP:
020 ike 0:ipsec-at-remote:5207: trans_id = KEY_IKE.
021 ike 0:ipsec-at-remote:5207: encapsulation = IKE/none
022 ike 0:ipsec-at-remote:5207: type=OAKLEY_ENCRYPT_ALG, val=AES_CBC.
023 ike 0:ipsec-at-remote:5207: type=OAKLEY_HASH_ALG, val=SHA.
024 ike 0:ipsec-at-remote:5207: type=AUTH_METHOD, val=PRESHARED_KEY.
025 ike 0:ipsec-at-remote:5207: type=OAKLEY_GROUP, val=1024.
026 ike 0:ipsec-at-remote:5207: ISKAMP SA lifetime=28800
027 ike 0:ipsec-at-remote:5207: selected NAT-T version: RFC 3947
028 ike 0:ipsec-at-remote:5207: cookie e24237b7f7d82b54/05958ceab9c7cb69
029 ike 0:ipsec-at-remote:5207: out lots-of-hex
030 ike 0:ipsec-at-remote:5207: sent IKE msg (ident_r1send): my-ip:500->hub-ip:500, len=144, id=e24237b7f7d82b54/05958ceab9c7cb69
031 ike 0: comes hub-ip:500->my-ip:500,ifindex=5....
032 ike 0: IKEv1 exchange=Identity Protection id=e24237b7f7d82b54/05958ceab9c7cb69 len=228
033 ike 0: in lots-of-hex
034 ike 0:ipsec-at-remote:5207: responder:main mode get 2nd message...
035 ike 0:ipsec-at-remote:5207: NAT detected: ME 
036 ike 0:ipsec-at-remote:5207: out lots-of-hex
037 ike 0:ipsec-at-remote:5207: sent IKE msg (ident_r2send): my-ip:500->hub-ip:500, len=228, id=e24237b7f7d82b54/05958ceab9c7cb69
038 ike 0:ipsec-at-remote:5207: ISAKMP SA e24237b7f7d82b54/05958ceab9c7cb69 key 16:C9C7F81FED3D466E35FCD4A0EAB97E7A
039 ike 0: comes hub-ip:4500->my-ip:4500,ifindex=5....
040 ike 0: IKEv1 exchange=Identity Protection id=e24237b7f7d82b54/05958ceab9c7cb69 len=108
041 ike 0: in lots-of-hex
042 ike 0:ipsec-at-remote:5207: responder: main mode get 3rd message...
043 ike 0:ipsec-at-remote:5207: dec E24237B7F7D82B5405958CEAB9C7CB6905100201000000000000006C080000140200000069707365632D746F2D6368770B0000187701753AA31881780512AC6B84CD46DC21F1AAB40000001C0000000101106002E24237B7F7D82B5405958CEAB9C7CB699F89A58D863BFD07
044 ike 0:ipsec-at-remote:5207: received notify type 24578
045 ike 0:ipsec-at-remote:5207: PSK authentication succeeded
046 ike 0:ipsec-at-remote:5207: authentication OK
047 ike 0:ipsec-at-remote:5207: enc lots-of-hex
048 ike 0:ipsec-at-remote:5207: port change 500 -> 4500
049 ike 0:ipsec-at-remote:5207: out lots-of-hex
050 ike 0:ipsec-at-remote:5207: sent IKE msg (ident_r3send): my-ip:4500->hub-ip:4500, len=76, id=e24237b7f7d82b54/05958ceab9c7cb69
051 ike 0:ipsec-at-remote:5207: established IKE SA e24237b7f7d82b54/05958ceab9c7cb69
052 ike 0:ipsec-at-remote:5207: processing INITIAL-CONTACT
053 ike 0:ipsec-at-remote: flushing 
054 ike 0:ipsec-at-remote: flushed 
055 ike 0:ipsec-at-remote:5207: processed INITIAL-CONTACT
056 ike 0:ipsec-at-remote: set oper up
057 ike 0:ipsec-at-remote: schedule auto-negotiate
058 ike 0:ipsec-at-remote:5207: no pending Quick-Mode negotiations
059 ike 0:ipsec-at-remote:ipsec-at-remote-p2: IPsec SA connect 5 my-ip->hub-ip:4500
060 ike 0:ipsec-at-remote:ipsec-at-remote-p2: using existing connection
061 ike 0:ipsec-at-remote:ipsec-at-remote-p2: config found
062 ike 0:ipsec-at-remote:ipsec-at-remote-p2: IPsec SA connect 5 my-ip->hub-ip:4500 negotiating
063 ike 0:ipsec-at-remote: carrier up
064 ike 0:ipsec-at-remote:5207: cookie e24237b7f7d82b54/05958ceab9c7cb69:6eab0c3a
065 ike 0:ipsec-at-remote:5207:ipsec-at-remote-p2:628914: natt flags 0x23, encmode 1->3
066 ike 0:ipsec-at-remote:5207:ipsec-at-remote-p2:628914: initiator selectors 0 0:0.0.0.0/0.0.0.0:0:0->0:0.0.0.0/0.0.0.0:0:0
067 ike 0:ipsec-at-remote:5207: enc lots-of-hex
068 ike 0:ipsec-at-remote:5207: out lots-of-hex
069 ike 0:ipsec-at-remote:5207: sent IKE msg (quick_i1send): my-ip:4500->hub-ip:4500, len=300, id=e24237b7f7d82b54/05958ceab9c7cb69:6eab0c3a
070 ike 0:ipsec-at-remote:5207: out lots-of-hex
071 ike 0:ipsec-at-remote:5207: sent IKE msg (P2_RETRANSMIT): my-ip:4500->hub-ip:4500, len=300, id=e24237b7f7d82b54/05958ceab9c7cb69:6eab0c3a
072 ike 0:ipsec-at-remote: link is idle 5 my-ip->hub-ip:4500 dpd=1 seqno=530cf
073 ike 0:ipsec-at-remote:5207: send IKEv1 DPD probe, seqno 340175
074 ike 0:ipsec-at-remote:5207: enc lots-of-hex
075 ike 0:ipsec-at-remote:5207: out lots-of-hex
076 ike 0:ipsec-at-remote:5207: sent IKE msg (R-U-THERE): my-ip:4500->hub-ip:4500, len=92, id=e24237b7f7d82b54/05958ceab9c7cb69:da5ac04c
077 ike 0:ipsec-at-remote:5207: out lots-of-hex
078 ike 0:ipsec-at-remote:5207: sent IKE msg (P2_RETRANSMIT): my-ip:4500->hub-ip:4500, len=300, id=e24237b7f7d82b54/05958ceab9c7cb69:6eab0c3a
079 ike 0:ipsec-at-remote:ipsec-at-remote-p2: IPsec SA connect 5 my-ip->hub-ip:4500
080 ike 0:ipsec-at-remote:ipsec-at-remote-p2: using existing connection
081 ike 0:ipsec-at-remote:ipsec-at-remote-p2: config found
082 ike 0: comes hub-ip:4500->my-ip:4500,ifindex=5....
083 ike 0: IKEv1 exchange=Identity Protection id=e24237b7f7d82b54/05958ceab9c7cb69 len=108
084 ike 0: in lots-of-hex
085 ike 0:ipsec-at-remote:5207: retransmission, re-send last message
086 ike 0:ipsec-at-remote:5207: out lots-of-hex
087 ike 0:ipsec-at-remote:5207: sent IKE msg (retransmit): my-ip:4500->hub-ip:4500, len=76, id=e24237b7f7d82b54/05958ceab9c7cb69
088 ike 0:ipsec-at-remote:ipsec-at-remote-p2: IPsec SA connect 5 my-ip->hub-ip:4500
089 ike 0:ipsec-at-remote:ipsec-at-remote-p2: using existing connection
090 ike 0:ipsec-at-remote:ipsec-at-remote-p2: config found
091 ike 0:ipsec-at-remote: link is idle 5 my-ip->hub-ip:4500 dpd=1 seqno=530cf
092 ike 0:ipsec-at-remote:5207: send IKEv1 DPD probe, seqno 340175
093 ike 0:ipsec-at-remote:5207: enc lots-of-hex
094 ike 0:ipsec-at-remote:5207: out lots-of-hex
095 ike 0:ipsec-at-remote:5207: sent IKE msg (R-U-THERE): my-ip:4500->hub-ip:4500, len=92, id=e24237b7f7d82b54/05958ceab9c7cb69:27ffdb1b
096 ike 0:ipsec-at-remote:ipsec-at-remote-p2: IPsec SA connect 5 my-ip->hub-ip:4500
097 ike 0:ipsec-at-remote:ipsec-at-remote-p2: using existing connection
098 ike 0:ipsec-at-remote:ipsec-at-remote-p2: config found
099 ike 0:ipsec-at-remote:5207: out lots-of-hex
100 ike 0:ipsec-at-remote:5207: sent IKE msg (P2_RETRANSMIT): my-ip:4500->hub-ip:4500, len=300, id=e24237b7f7d82b54/05958ceab9c7cb69:6eab0c3a
101 ike 0:ipsec-at-remote: link is idle 5 my-ip->hub-ip:4500 dpd=1 seqno=530cf
102 ike 0:ipsec-at-remote:5207: send IKEv1 DPD probe, seqno 340175
103 ike 0:ipsec-at-remote:5207: enc lots-of-hex
104 ike 0:ipsec-at-remote:5207: out lots-of-hex
105 ike 0:ipsec-at-remote:5207: sent IKE msg (R-U-THERE): my-ip:4500->hub-ip:4500, len=92, id=e24237b7f7d82b54/05958ceab9c7cb69:afd2468c
106 ike 0: comes hub-ip:4500->my-ip:4500,ifindex=5....
107 ike 0: IKEv1 exchange=Identity Protection id=e24237b7f7d82b54/05958ceab9c7cb69 len=108
108 ike 0: in lots-of-hex
109 ike 0:ipsec-at-remote:5207: retransmission, re-send last message
110 ike 0:ipsec-at-remote:5207: out lots-of-hex
111 ike 0:ipsec-at-remote:5207: sent IKE msg (retransmit): my-ip:4500->hub-ip:4500, len=76, id=e24237b7f7d82b54/05958ceab9c7cb69
112 ike 0:ipsec-at-remote:ipsec-at-remote-p2: IPsec SA connect 5 my-ip->hub-ip:4500
113 ike 0:ipsec-at-remote:ipsec-at-remote-p2: using existing connection
114 ike 0:ipsec-at-remote:ipsec-at-remote-p2: config found
115 ike 0:ipsec-at-remote: carrier down
116 ike 0:ipsec-at-remote: set oper down
117 ike 0:ipsec-at-remote: deleting
118 ike 0:ipsec-at-remote: flushing 
119 ike 0:ipsec-at-remote: flushed 
120 ike 0:ipsec-at-remote:5207: send ISAKMP delete e24237b7f7d82b54/05958ceab9c7cb69
121 ike 0:ipsec-at-remote:5207: enc lots-of-hex
122 ike 0:ipsec-at-remote:5207: out lots-of-hex
123 ike 0:ipsec-at-remote:5207: sent IKE msg (ISKAMP SA DELETE-NOTIFY): my-ip:4500->hub-ip:4500, len=92, id=e24237b7f7d82b54/05958ceab9c7cb69:94c120a8
124 ike 0:ipsec-at-remote: reset NAT-T
125 ike 0:ipsec-at-remote: deleted
126 ike 0:ipsec-at-remote: schedule auto-negotiate
127 ike 0:ipsec-at-remote: link fail 5 my-ip->hub-ip:4500 dpd=1
128 ike 0:ipsec-at-remote: ignoring since port is not 500
129 ike 0:ipsec-at-remote: reset NAT-T settings
130 ike shrank heap by 131072 bytes
131 ike 0:ipsec-at-remote: auto-negotiate connection
132 ike 0:ipsec-at-remote: created connection: 0x1d2a260 5 my-ip->hub-ip:500.
133 ike 0:ipsec-at-remote:5208: initiator: main mode is sending 1st message...
134 ike 0:ipsec-at-remote:5208: cookie 20a7e7d08c993331/0000000000000000
135 ike 0:ipsec-at-remote:5208: out lots-of-hex
136 ike 0:ipsec-at-remote:5208: sent IKE msg (ident_i1send): my-ip:500->hub-ip:500, len=244, id=20a7e7d08c993331/0000000000000000
137 ike 0:ipsec-at-remote:5208: out lots-of-hex
138 ike 0:ipsec-at-remote:5208: sent IKE msg (P1_RETRANSMIT): my-ip:500->hub-ip:500, len=244, id=20a7e7d08c993331/0000000000000000
139 ike 0: comes hub-ip:500->my-ip:500,ifindex=5....
140 ike 0: IKEv1 exchange=Identity Protection id=ada75449e872ca9a/0000000000000000 len=244
141 ike 0: in lots-of-hex
142 ike 0: found ipsec-at-remote my-ip 5 -> hub-ip:500
143 ike 0:ipsec-at-remote:5209: responder: main mode get 1st message...

Debug from the hub site - note, no mention of psk success

001 # diag vpn ike log-filter list 
002 vd: any
003 name: ipsec-at-hub
004 interface: any
005 IPv4 source: any
006 IPv4 dest: rem-nat-ip
007 IPv6 source: any
008 IPv6 dest: any
009 source port: any
010 dest port: any
011 
012 # diag deb app ike -1
013 # diag deb en
014 
015 
016 ike 0: comes rem-nat-ip:500->hub-ip:500,ifindex=8....
017 ike 0: IKEv1 exchange=Identity Protection id=0e28555e25dd2dd9/ab80a5ace2a1c17e len=144
018 ike 0: in lots-of-hex
019 ike 0:ipsec-at-hub:9054: initiator: main mode get 1st response...
020 ike 0:ipsec-at-hub:9054: VID RFC 3947 4A131C81070358455C5728F20E95452F
021 ike 0:ipsec-at-hub:9054: VID DPD AFCAD71368A1F1C96B8696FC77570100
022 ike 0:ipsec-at-hub:9054: DPD negotiated
023 ike 0:ipsec-at-hub:9054: VID FORTIGATE 8299031757A36082C6A621DE000402B1
024 ike 0:ipsec-at-hub:9054: peer is FortiGate/FortiOS (v4 b689)
025 ike 0:ipsec-at-hub:9054: selected NAT-T version: RFC 3947
026 ike 0:ipsec-at-hub:9054: negotiation result
027 ike 0:ipsec-at-hub:9054: proposal id = 1:
028 ike 0:ipsec-at-hub:9054:   protocol id = ISAKMP:
029 ike 0:ipsec-at-hub:9054:      trans_id = KEY_IKE.
030 ike 0:ipsec-at-hub:9054:      encapsulation = IKE/none
031 ike 0:ipsec-at-hub:9054:         type=OAKLEY_ENCRYPT_ALG, val=AES_CBC.
032 ike 0:ipsec-at-hub:9054:         type=OAKLEY_HASH_ALG, val=SHA.
033 ike 0:ipsec-at-hub:9054:         type=AUTH_METHOD, val=PRESHARED_KEY.
034 ike 0:ipsec-at-hub:9054:         type=OAKLEY_GROUP, val=1024.
035 ike 0:ipsec-at-hub:9054: ISKAMP SA lifetime=28800
036 ike 0:ipsec-at-hub:9054: out lots-of-hex
037 ike 0:ipsec-at-hub:9054: sent IKE msg (ident_i2send): hub-ip:500->rem-nat-ip:500, len=228, id=0e28555e25dd2dd9/ab80a5ace2a1c17e
038 ike 0: comes rem-nat-ip:500->hub-ip:500,ifindex=8....
039 ike 0: IKEv1 exchange=Identity Protection id=0e28555e25dd2dd9/ab80a5ace2a1c17e len=144
040 ike 0: in lots-of-hex
041 ike 0:ipsec-at-hub:9054: retransmission, re-send last message
042 ike 0:ipsec-at-hub:9054: out lots-of-hex
043 ike 0:ipsec-at-hub:9054: sent IKE msg (retransmit): hub-ip:500->rem-nat-ip:500, len=228, id=0e28555e25dd2dd9/ab80a5ace2a1c17e
044 ike 0: comes rem-nat-ip:500->hub-ip:500,ifindex=8....
045 ike 0: IKEv1 exchange=Identity Protection id=0e28555e25dd2dd9/ab80a5ace2a1c17e len=228
046 ike 0: in lots-of-hex
047 ike 0:ipsec-at-hub:9054: initiator: main mode get 2nd response...
048 ike 0:ipsec-at-hub:9054: NAT detected: PEER
049 ike 0:ipsec-at-hub:9054: NAT-T float port 4500
050 ike 0:ipsec-at-hub:9054: ISAKMP SA 0e28555e25dd2dd9/ab80a5ace2a1c17e key 16:D002ADFCCECD76D00CADE8B465395119
051 ike 0:ipsec-at-hub:9054: add INITIAL-CONTACT
052 ike 0:ipsec-at-hub:9054: enc lots-of-hex
053 ike 0:ipsec-at-hub:9054: out lots-of-hex
054 ike 0:ipsec-at-hub:9054: sent IKE msg (ident_i3send): hub-ip:4500->rem-nat-ip:4500, len=108, id=0e28555e25dd2dd9/ab80a5ace2a1c17e
055 ike 0: comes rem-nat-ip:500->hub-ip:500,ifindex=8....
056 ike 0: IKEv1 exchange=Identity Protection id=0e28555e25dd2dd9/ab80a5ace2a1c17e len=228
057 ike 0: in lots-of-hex
058 ike 0:ipsec-at-hub:9054: retransmission, re-send last message
059 ike 0:ipsec-at-hub:9054: out lots-of-hex
060 ike 0:ipsec-at-hub:9054: sent IKE msg (retransmit): hub-ip:4500->rem-nat-ip:4500, len=108, id=0e28555e25dd2dd9/ab80a5ace2a1c17e
061 ike 0:ipsec-at-hub:9054: out lots-of-hex
062 ike 0:ipsec-at-hub:9054: sent IKE msg (P1_RETRANSMIT): hub-ip:4500->rem-nat-ip:4500, len=108, id=0e28555e25dd2dd9/ab80a5ace2a1c17e
063 ike shrank heap by 122880 bytes
064 ike 0:ipsec-at-hub: NAT keep-alive 8 hub-ip->rem-nat-ip:4500.
065 ike 0:ipsec-at-hub:9054: negotiation timeout, deleting
066 ike 0:ipsec-at-hub: connection expiring due to phase1 down
067 ike 0:ipsec-at-hub: deleting
068 ike 0:ipsec-at-hub: flushing 
069 ike 0:ipsec-at-hub: flushed 
070 ike 0:ipsec-at-hub: reset NAT-T
071 ike 0:ipsec-at-hub: deleted
072 ike 0:ipsec-at-hub: schedule auto-negotiate
073 ike 0:ipsec-at-hub:9069: initiator: main mode is sending 1st message...
074 ike 0:ipsec-at-hub:9069: cookie da08c6777fc5160d/0000000000000000
075 ike 0:ipsec-at-hub:9069: out lots-of-hex
076 ike 0:ipsec-at-hub:9069: sent IKE msg (ident_i1send): hub-ip:500->rem-nat-ip:500, len=244, id=da08c6777fc5160d/0000000000000000
077 ike 0: comes rem-nat-ip:4500->hub-ip:500,ifindex=8....
078 ike 0:ipsec-at-hub:9069: out lots-of-hex
079 ike 0:ipsec-at-hub:9069: sent IKE msg (P1_RETRANSMIT): hub-ip:500->rem-nat-ip:500, len=244, id=da08c6777fc5160d/0000000000000000
080 ike 0: comes rem-nat-ip:4500->hub-ip:500,ifindex=8....
081 ike 0: comes rem-nat-ip:4500->hub-ip:500,ifindex=8....
082 ike 0:ipsec-at-hub:9069: out lots-of-hex
083 ike 0:ipsec-at-hub:9069: sent IKE msg (P1_RETRANSMIT): hub-ip:500->rem-nat-ip:500, len=244, id=da08c6777fc5160d/0000000000000000
084 ike 0:ipsec-at-hub:9069: negotiation timeout, deleting
085 ike 0:ipsec-at-hub: connection expiring due to phase1 down
086 ike 0:ipsec-at-hub: deleting
087 ike 0:ipsec-at-hub: flushing 
088 ike 0:ipsec-at-hub: flushed 
089 ike 0:ipsec-at-hub: deleted
090 ike 0:ipsec-at-hub: schedule auto-negotiate
091 ike 0:ipsec-at-hub:9085: initiator: main mode is sending 1st message...
092 ike 0:ipsec-at-hub:9085: cookie f7b7009c1f709f12/0000000000000000
093 ike 0:ipsec-at-hub:9085: out lots-of-hex
094 ike 0:ipsec-at-hub:9085: sent IKE msg (ident_i1send): hub-ip:500->rem-nat-ip:500, len=244, id=f7b7009c1f709f12/0000000000000000
095 ike 0:ipsec-at-hub:9085: out lots-of-hex
096 ike 0:ipsec-at-hub:9085: sent IKE msg (P1_RETRANSMIT): hub-ip:500->rem-nat-ip:500, len=244, id=f7b7009c1f709f12/0000000000000000
097 ike 0:ipsec-at-hub:9085: out lots-of-hex
098 ike 0:ipsec-at-hub:9085: sent IKE msg (P1_RETRANSMIT): hub-ip:500->rem-nat-ip:500, len=244, id=f7b7009c1f709f12/0000000000000000

Any suggestions

[emnoc]

I can tell by the   IKE SA your SAs are coming and going due to the secs in those outputs. But what is the link between the 2 FGT ( see my red arrow )

I'm trying to get an ideal of what your topology is and that link is not 100% clear for  me. Is the VPN tunnel terminated on FGT1 x.1  and  FGT2 x.2 or is it on FGT w.1 and FGT w.2 ? I'm confused.

On the next issues, what I would do ;

1: drop the  OSPF routing for now  this will add to more complex and t-shoot and diagnostics

2: configure the vpn tunnels as  static-route ( apply routes in each hub---> Spoke(s)  &  Spoke(s)------hub ) for now

3: actually concentrate on one vpn tunnel at a time ( disable the other spokes )

4: get one hub<>spoke vpn up and then apply the others

5: drop the stand-by FGT or swack to the stand-by but eliminate one FGT from the HA if possible

6: re-check your route table for any mis-rouetd or improper routes & next-hop(s)

BTW; VPN setups and t-shooting should not be this hard.

[journeyman]

Whatisthis? The link w.1 to w.2 is the primary link between FGT1 and FGT2. There is no ipsec on this link. It carries ospf with default cost whereas ospf cost is set high on the tunnel (when we, er, have a tunnel..).

The ipsec tunnel is from FGT1 x.1 to FGT2 y.1.

Along the lines of what you suggest, I intend to test on some spare hardware we have. That will involve configuring from scratch so we can see if the issue is present. Using our ipsec build template the tunnel should come up as soon as we have policies each end, before routing is configured.

I'm glad for your comment that ipsec shouldn't be this hard - so far it has been quite trouble free. I'm worried what silly snafu has yet to be discovered.

[emnoc]

Okay so the vpn is a backup link, what I would do;

1; place static routes on the hub to remote "peers address" and likewise. this ensue the ESP/IKE traffic for each peer goes out that  interface and not over the main.

e.g

router static

 edit 777

    set dst y.y.y.y2/32

    set gate  x.x.x.x2

2:DO NOT redistribute these statics into your  OSPF policy for the nexthops of the local router/3g-routers

3: enable the ipsec vpn tunnel & with no  ospf and with the correct fwpolicies

See if that comes up and if you diag vpn tunnel output shows SPI in/out

[journeyman]

Test setup: New test tunnel built from our normal template on test hardware running 4.3.18.

Tunnel comes up immediately relevant firewall policies are enabled.

From each FGT, exec ping <system.interface.ipsec-tunnel-interface.remote-ip> working fine shows tunnel is functioning. Ping times believable.

Configure HA on hub FGT (master only, no slave) - tunnel comes up again.

diag vpn ike gateway list looks good, status established with tunnel stable (age increasing).

diag vpn tunnel also looks good with spi.

Next week's trick will be to add / move the test remote site to the production FGT hub. This should be illuminating...

in progress..

[emnoc]

That's good news and I knew you can get it working. Now the other issues is your "opsf", you need to think about this and might want to include it in your test-env if possible.

If you enable  OSPF and things start to break,than you know what/where to go back & start from.

[journeyman]

All production tunnels disabled except for one (in case the wind changes, or something).

Migrate test remote site into production today, ipsec tunnel is established immediately and remains stable.

Compare ipsec phase1-interface and phase2-interface with non-working tunnels, no difference encountered.

Hrrm.

OSPF configuration added between the hub and test remote. OSPF neighbor established immediately (no routing info added yet).

Hrrm. Hrrm.

Production hub is still a standalone unit (can't swap to HA without outage notification process etc).

It is not at all obvious to me why a newly built site works but the existing ones do not.

ipsec config - check

interface config - check

firewall policies - check

routing config - check

I am tempted to - urggh - tear down and rebuild an existing tunnel as an experiment.

[emnoc]

Keep pecking at it, but if it's a VPN tunnel that can not establish, I would work them in one at a time and use diag debug flow for the peer (s) that don't comes up. Ensure thing  like NAT-T or recursive routing is not an issues. Ensure LAN LOCAL<--2--->REMOTE LAN routes are in place. I believe your not hitting any hardware limits  or anything else but if your doing this in a vdom make sure you don't have any global resources limits per-vdom that might effected you that you over looked.

e.g

config system vdom-property

   edit vdom <blahBlah>

     set ipsec-phase1 <i#s>

     set ipsec-phase2 <i#s>

end

The diag debug would really be your best friend in this case diag debug application ike along with the diagnostic outputs from the show commands.

[journeyman]

Have had other priorities to attend to since my last post.

Entirely deleted one of the tunnels at the hub end (removed from ospf, removed firewall policies, removed ph2 and ph1) and rebuilt it (create ph1, ph2, config interface, one policy, no ospf). Same as before (so now, test tunnel working, rebuilt tunnel not).

diag vpn ike gate list shows the tunnel always connecting and always created less than 30 seconds ago.

At the hub end diag debug flow was run on the rebuilt tunnel using filter addr <remote-ip>. The output was very boring along the lines of

trace_id=11 msg="vd-root received a packet(prot=17, hub-ip:500->rem-ip:500) from local."

trace_id=11 msg="Find an existing session, id-00008d90, oiginal direction."

trace_id=12 msg="vd-root received a packet(prot=17, rem-ip:500->hub-ip:500) from internal4."

trace_id=12 msg="Find an existing session, id-00008d90, reply direction."Some different source and destination ports eg 4500-500, 4500-4500 and session IDs, but nothing else in the trace.

Since this is flow through the kernel, wouldn't there be more for each packet? Using packet flow on a faulty firewall policy shows a lot more output / processing.

Whilst feeling experimental, I crossed the tunnels (sort of) so that the hub end test pointed to the existing remote. That didn't work either. That would imply the problem is at the existing remote end. But the NAT device and FGT configurations have been checked and again and look fine.

Is the flow filter of addr <peer-ip> useful? Is there a better filter?

Since the tunnel is always connecting and never lasts past 30s, Ede suggested ph2 was failing. But the ph2 config is very simple. What could cause that? What can I do to debug that more closely?

What's the difference between diag deb app ike -1 and diag deb app ike 255? I don't see anything useful in 255 output for instance

Still waiting to find my snafu..

[emnoc]

I still don't think your phase1 has established from all of the stuff you have shown and has nothing to do with phase2 but phase2 is dependent on phase1 for the obvious reasons.

Just a suggestion, have you tried using ikev2 on hub+peer(s)? try that on the peer ad hub routed interface that's failing and let us know if it comes up.

Have you double and I mean triple checked your proposals  on the phase1/2 & make them single proposals offering and double check  them? ( i think you have but check again )

Ken