ADVPN in FortiOS 7.0.10

[UnderscoresAndDashes]

        I am in the process of configuring a new hub for our ADVPN-BGP environment. When I we were using FortiOS 7.0.8 it work worked with very few issues, but now in 7.0.10 I run into nothing but issues. When I run a ping from spoke to spoke, the first attempt will give me 2 successful pings and then die. If do an exec router clear bgp all and clear the table, will get successful pings for about 10 seconds and then it dies again. 

So I run diagnose vpn ike log filter mdst-addr4 x.x.x.x y.y.y.y on each of the spokes and I when the pings are successful I see what I am supposed to see, but in the same ping the connection dies and I see "SA is not ready yet, drop". What really gets me, is if I create a static route on one side for the spoke I want to reach, and point it to the bgp ipsec tunnel, everything runs fine. Why???

Is this expected behavior or is this some weird nuance of FortiOS? 

[akristof]

Hello,

I would need to see couple of things. First, routing-table after shortcut is created and then the shortcut itself.

You can attach these:

show vpn ipsec phase1-interface

diag netlink interface list

get router info routing-table all

diag vpn tunnel list

diag vpn ike gateway list

Please share this output when the ping is failing. Also share with source/destination you are using .

[UnderscoresAndDashes]

I replaced Public IP Addresses with x.x.x.x
Spoke1
edit "EDW_ADVPN"
set interface "port1"
set peertype any
set net-device enable
set proposal aes256-sha256 aes256-sha1
set add-route disable
set auto-discovery-receiver enable
set remote-gw x.x.x.x

if=lo family=00 type=772 index=1 mtu=16436 link=0 master=0
ref=10 state=present fw_flags=0 flags=loopback

if=dummy0 family=00 type=1 index=2 mtu=1500 link=0 master=0
ref=7 state=present fw_flags=0 flags=broadcast noarp

if=port1 family=00 type=1 index=3 mtu=9001 link=0 master=0
ref=29 state=start present fw_flags=0 flags=up broadcast run multicast

if=port2 family=00 type=1 index=4 mtu=1500 link=0 master=0
ref=13 state=start present fw_flags=3800 flags=up broadcast run multicast

if=port3 family=00 type=1 index=5 mtu=1500 link=0 master=0
ref=13 state=start present fw_flags=3800 flags=up broadcast run multicast

if=root family=00 type=772 index=6 mtu=16436 link=0 master=0
ref=28 state=start present fw_flags=0 flags=up loopback run

if=naf.root family=00 type=65535 index=7 mtu=1500 link=0 master=0
ref=10 state=start present fw_flags=82000000 flags=up broadcast run noarp multicast

if=l2t.root family=00 type=512 index=8 mtu=1300 link=0 master=0
ref=11 state=start present fw_flags=0 flags=up p2p run noarp multicast

if=ssl.root family=00 type=65534 index=9 mtu=1500 link=0 master=0
ref=11 state=start present fw_flags=0 flags=up p2p run noarp multicast

if=fortilink family=00 type=1 index=10 mtu=1500 link=0 master=0
ref=43 state=start present no_carrier fw_flags=8000 flags=up broadcast master multicast

if=LoopBack family=00 type=65535 index=11 mtu=1500 link=0 master=0
ref=11 state=start present fw_flags=0 flags=up broadcast loopback run noarp multicast

if=EDW_ADVPN family=00 type=768 index=12 mtu=1420 link=0 master=0
ref=15 state=start present fw_flags=0 flags=up p2p run noarp multicast

if=vsys_ha family=00 type=772 index=13 mtu=16436 link=0 master=0
ref=15 state=start present fw_flags=0 flags=up loopback run

if=port_ha family=00 type=1 index=14 mtu=1496 link=0 master=0
ref=12 state=start present no_carrier fw_flags=0 flags=up broadcast multicast

if=vsys_fgfm family=00 type=772 index=15 mtu=16436 link=0 master=0
ref=19 state=start present fw_flags=0 flags=up loopback run

if=tun_fgfm family=00 type=65534 index=16 mtu=1492 link=0 master=0
ref=13 state=start present fw_flags=0 flags=up p2p run noarp multicast

[UnderscoresAndDashes]

Routing table for VRF=0
S* 0.0.0.0/0 [5/0] via 172.150.149.1, port1, [1/0]
S 10.0.0.0/8 [10/0] via EDW_ADVPN tunnel x.x.x.x, [1/0]
S 10.0.1.0/24 [10/0] via EDW_ADVPN tunnel x.x.x.x, [1/0]
B 10.0.4.0/24 [200/0] via 172.50.20.3 (recursive via EDW_ADVPN tunnel x.x.x.x), 06:42:33
B 10.0.5.0/24 [200/0] via 172.50.20.235 (recursive via EDW_ADVPN tunnel x.x.x.x), 01:23:58
B 10.0.9.0/24 [200/0] via 172.50.20.3 (recursive via EDW_ADVPN tunnel x.x.x.x), 06:42:33
B 10.0.10.0/24 [200/0] via 172.50.30.2 (recursive via EDW_ADVPN tunnel x.x.x.x), 06:44:24
B 10.0.21.0/24 [200/0] via 172.50.20.243 (recursive via EDW_ADVPN tunnel x.x.x.x), 06:44:24
B 10.0.23.0/24 [200/0] via 172.50.20.245 (recursive via EDW_ADVPN tunnel x.x.x.x), 06:44:24

S 172.50.0.0/16 [10/0] via EDW_ADVPN tunnel x.x.x.x, [1/0]
S 172.50.40.0/22 [5/0] via EDW_ADVPN tunnel x.x.x.x, [1/0]
S 172.50.40.1/32 [15/0] via EDW_ADVPN tunnel x.x.x.x, [1/0]
C 172.50.40.3/32 is directly connected, EDW_ADVPN
C 172.150.149.0/24 is directly connected, port1
C 172.150.150.0/24 is directly connected, port2
C 172.150.151.0/24 is directly connected, port3
B 172.254.1.0/28 [200/0] via 172.50.20.3 (recursive via EDW_ADVPN tunnel x.x.x.x), 06:42:33
B 172.254.20.0/28 [200/0] via 172.50.20.5 (recursive via EDW_ADVPN tunnel x.x.x.x), 06:44:24
S 192.168.0.0/16 [10/0] via EDW_ADVPN tunnel x.x.x.x, [1/0]
B 192.168.4.0/24 [200/0] via 172.50.11.250 (recursive via EDW_ADVPN tunnel x.x.x.x), 06:44:24
B 192.168.5.0/24 [200/0] via 172.50.21.170 (recursive via EDW_ADVPN tunnel x.x.x.x), 06:43:55
B 192.168.10.0/24 [200/0] via 172.50.11.250 (recursive via EDW_ADVPN tunnel x.x.x.x), 06:44:24

[UnderscoresAndDashes]

list all ipsec tunnel in vd 0
------------------------------------------------------
name=EDW_ADVPN ver=1 serial=1 172.150.149.106:4500->x.x.x.x:4500 tun_id=x.x.x.x tun_id6=::x.x.x.x dst_mtu=9001 dpd-link=on weight=1
bound_if=3 lgwy=static/1 tun=intf mode=auto/1 encap=none/568 options[0238]=npu create_dev frag-rfc role=primary accept_traffic=1 overlay_id=0

proxyid_num=1 child_num=0 refcnt=5 ilast=5 olast=5 ad=r/2
stat: rxp=6526 txp=6529 rxb=464104 txb=370558
dpd: mode=on-demand on=1 idle=20000ms retry=3 count=0 seqno=4
natt: mode=keepalive draft=32 interval=10 remote_port=4500
proxyid=EDW_ADVPN proto=0 sa=1 ref=3 serial=1 auto-negotiate adr
src: 0:0.0.0.0-255.255.255.255:0
dst: 0:0.0.0.0-255.255.255.255:0
SA: ref=3 options=1a003 type=00 soft=0 mtu=8926 expire=1550/0B replaywin=2048
seqno=c7b esn=0 replaywin_lastseq=00000c7b qat=0 rekey=0 hash_search_len=1
life: type=01 bytes=0/0 timeout=42902/43200
dec: spi=c054ce1e esp=aes key=32 c6043818607f8e9b7bc26e3f72d4885b15669d48e74a3d17ea2a406721d5bac8
ah=sha1 key=20 e60d4ca6ebb4c022b9f5c0c339b30c5b24c9b504
enc: spi=64ef7fe9 esp=aes key=32 c3b7236ca6dc528bd1978c0f0f4e7c3b5d6abf611e795d652518fa82c8a891d1
ah=sha1 key=20 24d85206c097818719648de1b1ff4135ffc61278
dec:pkts/bytes=6388/446704, enc:pkts/bytes=6388/602570
npu_flag=00 npu_rgwy=x.x.x.x npu_lgwy=172.150.149.106 npu_selid=0 dec_npuid=0 enc_npuid=0

diag vpn ike gateway list

vd: root/0
name: EDW_ADVPN
version: 1
interface: port1 3
addr: 172.150.149.106:4500 -> x.x.x.x:4500
tun_id: x.x.x.x/::x.x.x.x
remote_location: 0.0.0.0
network-id: 0
virtual-interface-addr: 172.50.40.3 -> 172.50.40.1
created: 84377s ago
nat: me
auto-discovery: 2 receiver
IKE SA: created 1/1 established 1/1 time 3080/3080/3080 ms
IPsec SA: created 1/2 established 1/2 time 30/1570/3110 ms

id/spi: 23 5dedf985c02a5c09/f6953392e917b26b
direction: initiator
status: established 84377-84374s ago = 3080ms
proposal: aes256-sha1
key: 7c504611eadd4604-9ca934b6155496c6-89d5f6449ca05fe4-1146707f4c1ae86c
lifetime/rekey: 86400/1725
DPD sent/recv: 00000000/00001616

[akristof]

Hello.

So far, so good. Now, I would like to see this when shortcut is created. Also, please tell me source and destination IP of the traffic. You can put all the outputs into text file and attach it here.