|
Network Setup:
< SPOKE> ------dialup----- < HUB1> -------S2S---- < HUB2>
Context on the setup:
- Spoke FortiGate is connected to the HUB1 FortiGate via a Dialup tunnel.
- HUB1 FortiGate is connected to HUB2 via Site to site IPSEC tunnel.
- ADVPN shortcut tunnel is formed between Spoke and HUB2 FortiGate.
- Configuration details for HUB1 and HUB2 are not explained in this article as it mainly focuses on the Proxy ID creation and traffic drop issues with ADVPN.
- Relevant firewall policies to and from for site to site tunnel and Dialup connections should be configured, this article focuses on explaining the proxy IDs creation on the ADVPN tunnel which causes issues so some basic configs that are understandable are skipped.
Configuration on Spoke: (To note).
config vpn ipsec phase1-interface
edit "Spoke_INET1"
set interface "port1"
set ike-version 2
set authmethod signature
set peertype any
set net-device enab
set mode-cfg enable
set proposal aes256-sha256
set negotiate-timeout 10
set dpd on-idle
set dhgrp 19
set idle-timeout enable
set idle-timeoutinterval 5
set auto-discovery-receiver enable
set auto-discovery-shortcuts dependent
set network-overlay enable
set network-id 11
set remote-gw 110.20.30.1
end
Phase2:
config vpn ipsec phase2-interface
edit "Spoke_INET1"
set phase1name "Spoke_INET1"
set proposal aes256-sha256
set dhgrp 19
set auto-negotiate enable
set src-addr-type name
set dst-addr-type name
set keylifeseconds 3600
set src-name "LAN_Subnets" <--
set dst-name "all"
next
end
Based on this setup, the Spoke and HUB2 will form an ADVPN shortcut but the traffic will get dropped as per below.
Debug flow on Spoke FortiGate: (The ping is initiated from HUB2 to Spoke).
Spoke FGT # id=65308 trace_id=227 func=print_pkt_detail line=5857 msg="vd-root:0 received a packet(proto=1, 10.60.20.2:26->10.251.0.1:2048) tun_id=169.254.128.24 from Spoke_INET1_0. type=8, code=0, id=26, seq=0."
id=65308 trace_id=227 func=init_ip_session_common line=6043 msg="allocate a new session-00005dd7, tun_id=169.254.128.24"
id=65308 trace_id=227 func=iprope_dnat_check line=5302 msg="in-[Spoke_INET1_0], out-[]"--------------->Shortcut tunnel
id=65308 trace_id=227 func=iprope_dnat_tree_check line=824 msg="len=0"
----------------------
id=65308 trace_id=228 func=print_pkt_detail line=5857 msg="vd-root:0 received a packet(proto=1, 10.251.0.1:26->10.60.20.2:0) tun_id=0.0.0.0 from local. type=0, code=0, id=26, seq=0."
id=65308 trace_id=228 func=resolve_ip_tuple_fast line=5945 msg="Find an existing session, id-00005dd7, reply direction"
id=65308 trace_id=228 func=ip_session_core_in line=6559 msg="dir-1, tun_id=169.254.128.24"
id=65308 trace_id=228 func=ipsecdev_hard_start_xmit line=662 msg="enter IPSec interface Spoke_INET1_0, tun_id=169.254.128.24"
id=65308 trace_id=228 func=_do_ipsecdev_hard_start_xmit line=222 msg="output to IPSec tunnel Spoke_INET1_0 vrf 0"
id=65308 trace_id=228 func=ipsec_common_output4 line=789 msg="SA is not ready yet, drop"
Verification and Analysis.
IKE DEBUG FROM SPOKE:
ike 0:SPOKE_INET1_0:29:SPOKE_INET1:42501: IPsec SA selectors #src=17 #dst=1
ike 0:SPOKE_INET1_0:29:SPOKE_INET1:42501: src 0 7 0:10.24.5.0-10.24.5.255:0
ike 0:SPOKE_INET1_0:29:SPOKE_INET1:42501: src 1 7 0:10.62.64.96-10.62.64.127:0
ike 0:SPOKE_INET1_0:29:SPOKE_INET1:42501: src 2 7 0:10.102.48.32-10.102.48.63:0
ike 0:SPOKE_INET1_0:29:SPOKE_INET1:42501: src 3 7 0:10.102.48.96-10.102.48.127:0
ike 0:SPOKE_INET1_0:29:SPOKE_INET1:42501: src 4 7 0:10.102.48.128-10.102.48.255:0
ike 0:SPOKE_INET1_0:29:SPOKE_INET1:42501: src 5 7 0:10.102.49.0-10.102.49.255:0
ike 0:SPOKE_INET1_0:29:SPOKE_INET1:42501: src 6 7 0:10.102.50.0-10.102.51.255:0
ike 0:SPOKE_INET1_0:29:SPOKE_INET1:42501: src 7 7 0:10.102.52.0-10.102.55.255:0
ike 0:SPOKE_INET1_0:29:SPOKE_INET1:42501: src 8 7 0:10.102.56.0-10.102.57.255:0
ike 0:SPOKE_INET1_0:29:SPOKE_INET1:42501: src 9 7 0:10.102.58.0-10.102.59.255:0
ike 0:SPOKE_INET1_0:29:SPOKE_INET1:42501: src 10 7 0:10.102.60.0-10.102.61.255:0
ike 0:SPOKE_INET1_0:29:SPOKE_INET1:42501: src 11 7 0:10.131.43.0-10.131.43.255:0
ike 0:SPOKE_INET1_0:29:SPOKE_INET1:42501: src 12 7 0:172.18.10.32-172.18.10.32:0
ike 0:SPOKE_INET1_0:29:SPOKE_INET1:42501: src 13 7 0:10.102.48.0-10.102.48.31:0
ike 0:SPOKE_INET1_0:29:SPOKE_INET1:42501: src 14 7 0:10.102.48.64-10.102.48.95:0
ike 0:SPOKE_INET1_0:29:SPOKE_INET1:42501: src 15 7 0:10.130.227.32-10.130.227.63:0
ike 0:SPOKE_INET1_0:29:SPOKE_INET1:42501: src 16 7 0:10.251.0.0-10.251.0.255:0
ike 0:SPOKE_INET1_0:29:SPOKE_INET1:42501: dst 0 7 0:0.0.0.0-255.255.255.255:0
ike 0:SPOKE_INET1_0:29:SPOKE_INET1:42501: add dynamic IPsec SA selectors
ike 0:SPOKE_INET1_0:29:SPOKE_INET1:42501: added dynamic IPsec SA proxyids, existing serial 3
ike 0:SPOKE_INET1_0:29:SPOKE_INET1:42501: add IPsec SA: SPIs=f8bf7b27/ab21abb9
ike 0:SPOKE_INET1_0:29:SPOKE_INET1:42501: IPsec SA dec spi f8bf7b27 key 32:44A7F071C2F394AEA0F787C41B3C2B4AE3338E5AFE1B954C4BBDD21EA57DC88D auth 32:8A674D1D3BB7378A1FCD50633
FD7F3D12594FAA96822F2E2211D8DFF90D6B773
ike 0:SPOKE_INET1_0:29:SPOKE_INET1:42501: IPsec SA enc spi ab21abb9 key 32:FD6EB5FF214E895864D707F8D32F52378AC206127FC5AEBCD3D2133812B5F3CB auth 32:BC65080207F32D066F91AA8FE
4AAE3A261CB90D4D5A5FE94760938613DC6CAC4
ike 0:SPOKE_INET1_0:29:SPOKE_INET1:42501: added IPsec SA: SPIs=f8bf7b27/ab21abb9
ike 0:SPOKE_INET1_0:SPOKE_INET1: IPsec SA connect 3 110.20.30.2->110.20.30.3:0
ike 0:SPOKE_INET1_0:SPOKE_INET1: using existing connection
ike 0:SPOKE_INET1_0:SPOKE_INET1: traffic triggered, serial=1 1:10.251.0.1:0->1:10.60.20.2:0
ike 0:SPOKE_INET1:SPOKE_INET1: config found
ike 0:SPOKE_INET1_0:SPOKE_INET1: IPsec SA connect 3 110.20.30.2->110.20.30.3:500 negotiating
ike 0:SPOKE_INET1_0:29:42502 initiating CREATE_CHILD exchange
ike 0:SPOKE_INET1_0:29:SPOKE_INET1:42502: PFS enabled
ike 0:SPOKE_INET1_0:29:SPOKE_INET1:42502: generate DH public value request queued
ike 0:SPOKE_INET1_0:29:42502 initiating CREATE_CHILD exchange
ike 0:SPOKE_INET1_0:29:SPOKE_INET1:42502: PFS enabled
ike 0:SPOKE_INET1_0:29: enc 280000340000003001030404F8BF7B280300000C0100000C800E0100030000080300000C03000008040000130000000805000000220000149484C1845469EC1026F515EF72F190
252C00004800130000EF92AC7779493EDE0DDCF7F2126DE2EC110E70BE054CF187D5546C777562EDD00364C45D1AA653D25EE3A675F374C2A315BF5346E79C70ABF5E317358CCB6FE42D0001481400000007000
0100000FFFF0AFB00010AFB0001070000100000FFFF0A1805000A1805FF070000100000FFFF0A3E40600A3E407F070000100000FFFF0A6630200A66303F070000100000FFFF0A6630600A66307F070000100000
------------------------------
ike 0:SPOKE_INET1_0:29: out 55270852A5B82DE41151451D626FFF2C2E202400000000080000025021000234D115C638038AC0746FE136D7699F590BD3F7B65CFEE96ED5CF41C8ED428FB4E4B4FD6B12B298D6
052CBCF84780275554A40765583CCCDB4EA6023C0A2CE34082242BCEF5E93F44646CB0CE2F871048992D8C8BAC7C100655D29066B34CC791C1B1FAA97F6040369F16D1B34C1C0EF871AB5CA3EE5A94C4820C2E5
----------------------
ike 0:SPOKE_INET1_0:29: sent IKE msg (CREATE_CHILD): 110.20.30.2:500->110.20.30.3:500, len=592, vrf=0, id=55270852a5b82de4/1151451d626fff2c:00000008
ike 0: comes 110.20.30.3:500->110.20.30.2:500,ifindex=3,vrf=0....
ike 0: IKEv2 exchange=CREATE_CHILD_RESPONSE id=55270852a5b82de4/1151451d626fff2c:00000008 len=544
----------------------------------------------
ike 0:SPOKE_INET1_0:29:SPOKE_INET1:42502: src 15 7 0:10.130.227.32-10.130.227.63:0
ike 0:SPOKE_INET1_0:29:SPOKE_INET1:42502: src 16 7 0:10.251.0.0-10.251.0.255:0
ike 0:SPOKE_INET1_0:29:SPOKE_INET1:42502: dst 0 7 0:0.0.0.0-255.255.255.255:0
ike 0:SPOKE_INET1_0:29:SPOKE_INET1:42502: add dynamic IPsec SA selectors
ike 0:SPOKE_INET1_0:29:SPOKE_INET1:42502: added dynamic IPsec SA proxyids, existing serial 3
ike 0:SPOKE_INET1_0:29:SPOKE_INET1:42502: add IPsec SA: SPIs=f8bf7b28/ab21abba
Here traffic was triggered for Serial 1 but the SA is added on Serial 3 and this will cause the traffic to fail/drop.
- The below command can show the same:
diagnose vpn tunnel list
name=_INET1_0 ver=2 serial=c 110.20.30.2:0->110.20.30.3:0 tun_id=169.254.128.24 tun_id6=::10.0.0.11 dst_mtu=1500 dpd-link=on weight=1
proxyid=!SPOKE_INET1 proto=1 sa=0 ref=1 serial=2 adr
src: 1:169.254.11.5-169.254.11.5:0
dst: 1:0.0.0.0-255.255.255.255:0
proxyid=SPOKE_INET1 proto=0 sa=0 ref=2 serial=1 auto-negotiate adr
src: 0:10.24.5.0-10.24.5.255:0 0:10.62.64.96-10.62.64.127:0 0:10.102.48.32-10.102.48.63:0 0:10.102.48.96-10.102.48.127:0 0:10.102.48.128-10.102.48.255:0 0:10.102.49.0-10.102.49.255:0 0:10.102.50.0-10.102.51.255:0 0:10.102.52.0-10.102.55.255:0 0:10.102.56.0-10.102.57.255:0 0:10.102.58.0-10.102.59.255:0 0:10.102.60.0-10.102.60.255:0 0:10.102.61.0-10.102.61.255:0 0:10.131.43.0-10.131.43.255:0 0:172.18.10.32-172.18.10.32:0 0:10.102.48.0-10.102.48.31:0 0:10.102.48.64-10.102.48.95:0 0:10.130.227.32-10.130.227.63:0 0:10.251.0.0-10.251.0.255:0 0:10.102.60.0-10.102.61.255:0
dst: 0:0.0.0.0-255.255.255.255:0
proxyid=SPOKE_INET1 proto=0 sa=4 ref=5 serial=3 auto-negotiate add-route adr
src: 0:10.24.5.0-10.24.5.255:0 0:10.62.64.96-10.62.64.127:0 0:10.102.48.32-10.102.48.63:0 0:10.102.48.96-10.102.48.127:0 0:10.102.48.128-10.102.48.255:0 0:10.102.49.0-10.102.49.255:0 0:10.102.50.0-10.102.51.255:0 0:10.102.52.0-10.102.55.255:0 0:10.102.56.0-10.102.57.255:0 0:10.102.58.0-10.102.59.255:0 0:10.102.60.0-10.102.61.255:0 0:10.131.43.0-10.131.43.255:0 0:172.18.10.32-172.18.10.32:0 0:10.102.48.0-10.102.48.31:0 0:10.102.48.64-10.102.48.95:0 0:10.130.227.32-10.130.227.63:0 0:10.251.0.0-10.251.0.255:0
dst: 0:0.0.0.0-255.255.255.255:0
SA: ref=3 options=3a203 type=00 soft=0 mtu=1438 expire=3371/0B replaywin=2048
seqno=1 esn=0 replaywin_lastseq=00000000 qat=0 rekey=0 hash_search_len=1
life: type=01 bytes=0/0 timeout=3591/3600
dec: spi=f8bf7b3b esp=aes key=32 e917e7c01fdeb5736123872d0f7fbb905bea2801d3d1943082b704e1c5f03a3d
ah=sha256 key=32 1cc0c3abdf96df7379c3007abed590c30d1915b575e3e8c00b9418da4de25273
enc: spi=ab21abcb esp=aes key=32 d2019fa1e0dd3cf7bb74e409d4c46a58533ccecfa3c054d41bbf19ecd46d4514
ah=sha256 key=32 c27641278196e73867ccdae77b2cd095f7819300b472b54b86435ffa1687138c
dec:pkts/bytes=0/0, enc:pkts/bytes=0/0
npu_flag=00 npu_rgwy=110.20.30.3 npu_lgwy=110.20.30.2 npu_selid=29 dec_npuid=0 enc_npuid=0
SA: ref=3 options=3a203 type=00 soft=0 mtu=1438 expire=3366/0B replaywin=2048
seqno=1 esn=0 replaywin_lastseq=00000002 qat=0 rekey=0 hash_search_len=1
life: type=01 bytes=0/0 timeout=3587/3600
dec: spi=f8bf7b3a esp=aes key=32 32340e0f0c25c422c9e01efbc7b53cae878e85f519759eacce14f7a0e4f54a07
ah=sha256 key=32 4777786921392f2d9cea52250f4a779fd2d4d9b14f72913956da3856e0622bcf
enc: spi=ab21abca esp=aes key=32 9a05756ce8256e1176c4c4e687c8abffdf257ce3bf8eb50a95e4b10ab0019066
ah=sha256 key=32 65751055eb20a198a5faaf411971d93015ff050c6fa904eb92f084d6ff0c5173
dec:pkts/bytes=2/168, enc:pkts/bytes=0/0
npu_flag=00 npu_rgwy=110.20.30.3 npu_lgwy=110.20.30.2 npu_selid=29 dec_npuid=0 enc_npuid=0
SA: ref=3 options=3a203 type=00 soft=0 mtu=1438 expire=3369/0B replaywin=2048
seqno=1 esn=0 replaywin_lastseq=00000000 qat=0 rekey=0 hash_search_len=1
life: type=01 bytes=0/0 timeout=3591/3600
----------------------------------------------------
Checking the Spoke FortiGate Phase2 config:
- There are overlapping subnets in it.
Address object: Source LAN_Subnets members:
10.102.48.0_27
10.102.48.128_25
10.102.48.32_27
10.102.48.64_27
10.102.48.96_27
10.102.49.0_24
10.102.50.0_23
10.102.52.0_22
10.102.56.0_23
10.102.58.0_23
10.102.60.0_23
10.102.60.0_24
10.102.61.0_24
10.130.227.32_27
10.131.43.0_24
10.24.5.0_24
10.62.64.96_27
- The solution for this would be to avoid using overlapping subnets on the source phase2 selectors and this is an expected behavior.
- Use subnets which will not overlap or use 0.0.0.0.0 to mitigate the issue.
Output is taken on Spoke2 after using source selectors as 0.0.0.0.0:
ike 0:SPOKE_INET1_0: shortcut selector added, new serial 1
ike 0:SPOKE_INET1_0: shortcut selector added, new serial 2
ike 0:SPOKE_INET1_0:34: check peer route: if_addr4_rcvd=1, if_addr6_rcvd=0, mode_cfg=0
ike 0:SPOKE_INET1_0:34: update peer route 0.0.0.0 -> 169.254.128.24
ike 0:SPOKE_INET1_0: add connected route 169.254.11.5 -> 169.254.128.24
ike 0:SPOKE_INET1_0:34: processing INITIAL-CONTACT
ike 0:SPOKE_INET1_0: flushing
ike 0:SPOKE_INET1_0: flushed
ike 0:SPOKE_INET1_0:34: processed INITIAL-CONTACT
ike 0:SPOKE_INET1: schedule auto-negotiate
ike 0:SPOKE_INET1_0:34: local cert, subject='FGVM01TM21001366', issuer='fortinet-subca2001'
ike 0:SPOKE_INET1_0:34: local CA cert, subject='fortinet-subca2001', issuer='fortinet-ca2'
ike 0:SPOKE_INET1_0:34: local CA cert, subject='fortinet-ca2', issuer='fortinet-ca2'
ike 0:SPOKE_INET1_0:34: add INTERFACE-ADDR4 169.254.11.5
ike 0:SPOKE_INET1_0:34:SPOKE_INET1:42897: replay protection enabled
ike 0:SPOKE_INET1_0:34:SPOKE_INET1:42897: set sa life soft seconds=3591.
ike 0:SPOKE_INET1_0:34:SPOKE_INET1:42897: set sa life hard seconds=3600.
ike 0:SPOKE_INET1_0:34:SPOKE_INET1:42897: IPsec SA selectors #src=1 #dst=1
ike 0:SPOKE_INET1_0:34:SPOKE_INET1:42897: src 0 7 0:0.0.0.0-255.255.255.255:0
ike 0:SPOKE_INET1_0:34:SPOKE_INET1:42897: dst 0 7 0:0.0.0.0-255.255.255.255:0
ike 0:SPOKE_INET1_0:34:SPOKE_INET1:42897: add dynamic IPsec SA selectors
ike 0:SPOKE_INET1_0:34:SPOKE_INET1:42897: added dynamic IPsec SA proxyids, existing serial 1
diagnose vpn tunnel list
name=SPOKE_INET1_0 ver=2 serial=e 110.20.30.2:0->110.20.30.3:0 tun_id=169.254.128.24 tun_id6=::10.0.0.13 dst_mtu=1500 dpd-link=on weight=1
bound_if=3 lgwy=static/1 tun=intf mode=dial_inst/3 encap=none/66232 options[102b8]=npu create_dev rgwy-chg frag-rfc role=primary accept_traffic=1 overlay_id=11
parent=SPOKE_INET1 index=0
proxyid_num=2 child_num=0 refcnt=6 ilast=0 olast=0 ad=r/2
stat: rxp=5 txp=5 rxb=420 txb=420
dpd: mode=on-idle on=1 idle=3000ms retry=3 count=0 seqno=10
natt: mode=none draft=0 interval=0 remote_port=0
fec: egress=0 ingress=0
proxyid=!SPOKE_INET1 proto=1 sa=0 ref=1 serial=2 adr
src: 1:169.254.11.5-169.254.11.5:0
dst: 1:0.0.0.0-255.255.255.255:0
proxyid=SPOKE_INET1 proto=0 sa=1 ref=3 serial=1 auto-negotiate adr
src: 0:0.0.0.0-255.255.255.255:0
dst: 0:0.0.0.0-255.255.255.255:0
SA: ref=3 options=3a203 type=00 soft=0 mtu=1438 expire=3532/0B replaywin=2048
seqno=6 esn=0 replaywin_lastseq=00000006 qat=0 rekey=0 hash_search_len=1
life: type=01 bytes=0/0 timeout=3591/3600
dec: spi=f8bf7b41 esp=aes key=32 235cbb9f7d7ddd735ce0bb7167567dcbce3b620ed7b35a40624caa37a09b20df
ah=sha256 key=32 0659fa8d4ada4b4df5f1951d7405d879de7bd021be1b486c77d9294de9e2dc76
enc: spi=ab21abcf esp=aes key=32 eced2e6e7db70b1ba0efbc3efc6ae43b87b2f933f8f4b716cbe99b6f84014b76
ah=sha256 key=32 9205ab6b798b3817fcd2b92e2e5790c0e87c7022f8fd387615472fc4c31cb382
dec:pkts/bytes=10/840, enc:pkts/bytes=10/1200
npu_flag=00 npu_rgwy=110.20.30.3 npu_lgwy=110.20.30.2 npu_selid=2f dec_npuid=0 enc_npuid=0
- On the above output, the SA gets added to Serial 1 as the traffic was triggered on it and this is an expected behavior and as a result, the traffic will work as well.
This article is relevant mainly for large scale ADVPN scenarios as described in this document:
SD-WAN in large scale deployments
|
