Support Forum
The Forums are a place to find answers on a range of Fortinet products from peers and product experts.
opt
New Contributor II

IPSEC tunnels behind CGNAT Starlink

Hi all. We have one very interesting case. We using Fortigate HA routers on HQ and Branch.

Branch is connected to HQ via 2 providers over IPSEC-SD-WAN tunnels.

But now we have often problems with these 2 providers availibility and decided to try Starlink.

We have connected Starlink router to Fortigate, switched Starlink router to bypas mode. 

Now Branch's Fortigate behind Starlink's CGNAT with IP 100.122.N.N 255.192.0.0 and we can't connect classic peer-to-peer IPSEC as before with those 2 providers with public ip on both sides.

 

So the question is how to make connection between HQ and branches?

We tried configure IPSEC with dilaup user on HQ side as listener and remote side connect to HQ public ip.

The tunnel become UP but there is no traffic between routers. I sugges that there is some configuration mistakes, but need more experience to debug it.

1 Solution
opt
New Contributor II

The link from Starlink should be connected directly to FGT port. When link connected thru switch's vlan strange things happening - ICMP work but other traffic not flow.

Found some information on Starlink's support page:

Can I use a network switch with Starlink?

Yes, you are welcome to connect your own equipment to Starlink. However, we cannot guarantee Starlink performance or compatibility with third party networking devices.

The case is closed because now router behind Starlink is connecting as dialup ipsec client to Fortigate with NAT-T.

View solution in original post

11 REPLIES 11
jintrah_FTNT
Staff
Staff

hi,

 

If tunnel is up, then negotiations are fine. But if traffic is not working, you should check for esp traffic between peers.

 

#diag sniff packe any 'host <remote peer IP> and esp' 6

 

best regards,

Jin

opt
New Contributor II

diagnose sniffer packet any "host 145.224.100.252 and esp" 6
interfaces=[any]
filters=[host 145.224.100.252 and esp]
^C
0 packets received by filter
0 packets dropped by kernel

francelottores
New Contributor

One of the main reasons Port Forwarding can be a problem on Starlink is that many ISPs use Carrier-Grade NAT (CGNAT) to conserve IP addresses. France lotto results

syordanov
Staff
Staff

Dear opt,

 

Please follow the suggestion of my colleague, see if ESP packets are sent out your exit interface.

If they are sent but nothing is received on other and maybe Starlink is doing filtering on the ESP packets. You can give a try with NAT-T to "forced".

 

Best regards,

 

Fortinet community

.
opt
New Contributor II

We have tried NAT-T on both side in forced mode, but nothing changed, ESP packets 0 received.

syordanov

Hello Opt,

 

Ok in that case you can try to see which is the public IP address assigned by Starlink or address used for SNAT to leave their network and go to internet.

Run this command on your FG(spoke)

# diagnose sys waninfo ipify port1 <--- replace port1 with interface configured for your VPN configuration

 

When you get the IP address from command above, you can run a sniffer on your HUB for this IP address something like diagnose sniffer packet any "host x.x.x.x" 4 , where x.x.x.x is the IP address from diagnose sys waninfo ipify , because it seams phase-2 is down and maybe phase-1 is down as well . This will give you more information what is send out / received for the phase-1 / phase-2 traffic.

 

Best regards,

 

Fortinet

.
opt
New Contributor II

#diagnose sys waninfo ipify port
Try to get my public IP through https://api.ipify.org with src_ip=0.0.0.0 device=port2 vfid=0(root) ...

Public/WAN IP: 145.224.100.252
Location:
Latitude: 52.229679
Longitude: 21.012230
Accuracy radius: Unknown
Time zone: Europe/Warsaw
City: Warsaw
Subdivisions:
0: Masovian
Country: Poland
Postal:
Code: 00-693
Continent: Europe
Registered country: Unknown
ISP: Unknown

some packets are runing thru interface

2022-10-18 14:56:46.919836 wan in 145.224.100.252.11905 -> <hub ip>.500: udp 108
2022-10-18 14:56:46.920049 wan out <hub ip>.500 -> 145.224.100.252.11905: udp 108
2022-10-18 14:57:06.991564 wan in 145.224.100.252.11905 -> <hub ip>.500: udp 108
2022-10-18 14:57:06.991804 wan out <hub ip>.500 -> 145.224.100.252.11905: udp 108

All the time repeating this messages

syordanov

Hello opt,
 
Thanks, so your HUB receives the IKE (UDP 500) messages from spoke which uses Starlink.
You can use the following commands to see if there is established phase1/phase2:
# diagnose  vpn ike gateway list 
# diagnose vpn tunnel list


For IKE debug you can run the following :

# diagnose debug reset
# diagnose vpn ike log filter name name_of_phase_1
# diagnose debug app ike -1
# diagnose debug console timestamp enable
# diagnose debug enable


Please do not forget to check if you have FW rules and static route for the VPN traffic from spoke to HUB, also you can run debug flow to see why traffic is not sent out the VPN interface on your spoke :

diagnose debug reset

diagnose debug flow filter saddr XXXXXX

diagnose debug flow filter daddr XXXXXX

Diagnose debug flow filter 

diag debug flow show function-name enable

diag debug flow show iprope enable

diagnose debug console timestamp enable

diagnose debug flow trace start 999

diagnose debug enable

 

Best regards,

 

Fortinet

 

.
opt
New Contributor II

Hi, all phase 1/2 established 

Below all debug logs from both sides.

 

First from spoke side as you see there is ip behind NAT 100.122.10.72:500

and second on hub side we get packets from public IP and port  1046 145.224.100.83:1046

 

spoke # dia vpn ike gateway list

vd: root/0
name: Connect-Dial-UP
version: 1
interface: port2 10
addr: 100.122.10.72:500 -> <HUB IP>:500
virtual-interface-addr: 10.62.10.1 -> 10.62.10.2
created: 560s ago
IKE SA: created 1/1 established 1/1 time 21350/21350/21350 ms
IPsec SA: created 1/1 established 1/1 time 100/100/100 ms

id/spi: 7 905126ead586fa65/c8fedf75665575f7
direction: initiator
status: established 560-539s ago = 21350ms
proposal: des-md5
key: 50fb3d8e88d8595f
lifetime/rekey: 86400/85560
DPD sent/recv: 000000a6/00000000

spoke # dia vpn tunn list
list all ipsec tunnel in vd 0
------------------------------------------------------
name=Connect-Dial-UP ver=1 serial=2 100.122.10.72:0-><HUB IP>:0 dst_mtu=1500
bound_if=10 lgwy=static/1 tun=intf/0 mode=auto/1 encap=none/520 options[0208]=npu frag-rfc run_state=0 accept_traffic=1 overlay_id=0

proxyid_num=1 child_num=0 refcnt=15 ilast=10 olast=10 ad=/0
stat: rxp=0 txp=5 rxb=0 txb=420
dpd: mode=on-idle on=1 idle=20000ms retry=3 count=0 seqno=168
natt: mode=none draft=0 interval=0 remote_port=0
proxyid=Connect-Dial-UP proto=0 sa=1 ref=2 serial=2
src: 0:0.0.0.0/0.0.0.0:0
dst: 0:0.0.0.0/0.0.0.0:0
SA: ref=5 options=10226 type=00 soft=0 mtu=1446 expire=42355/0B replaywin=2048
seqno=6 esn=0 replaywin_lastseq=00000000 itn=0 qat=0 hash_search_len=1
life: type=01 bytes=0/0 timeout=42899/43200
dec: spi=3ee3ff8d esp=des key=8 3a54253a29887249
ah=md5 key=16 14f51ad99e4c8afbb202db6e2ed5e68b
enc: spi=75712a5b esp=des key=8 f8d14d3a90643f03
ah=md5 key=16 527a434c60ac1b0e0a947c767e4cafc1
dec:pkts/bytes=0/0, enc:pkts/bytes=5/680
npu_flag=01 npu_rgwy=<HUB IP> npu_lgwy=100.122.10.72 npu_selid=3 dec_npuid=0 enc_npuid=1
run_tally=1

spoke #
spoke # diagnose debug flow filter saddr 100.122.10.72

spoke # diagnose debug flow filter daddr <HUB IP>

spoke # diagnose debug flow filter
vf: any
proto: any
Host addr: any
host saddr: 100.122.10.72-100.122.10.72
host daddr: <HUB IP>-<HUB IP>
port: any
sport: any
dport: any

spoke # dia debug flow show function-name enable
show function name

spoke # dia debug flow show iprope enable
show trace messages about iprope

spoke # diagnose debug console timestamp enable

spoke # dia debug flow trace start 999

spoke # dia debug enable

spoke # 2022-10-19 14:08:57 id=20085 trace_id=2 func=print_pkt_detail line=5727 msg="vd-root:0 received a packet(proto=17, 100.122.10.72:500-><HUB IP>:500) from local. "
2022-10-19 14:08:57 id=20085 trace_id=2 func=resolve_ip_tuple_fast line=5808 msg="Find an existing session, id-00022a0a, original direction"
2022-10-19 14:08:57 id=20085 trace_id=2 func=ipd_post_route_handler line=490 msg="out port2 vwl_zone_id 0, state2 0x0, quality 0.
"
2022-10-19 14:09:18 id=20085 trace_id=3 func=print_pkt_detail line=5727 msg="vd-root:0 received a packet(proto=17, 100.122.10.72:500-><HUB IP>:500) from local. "
2022-10-19 14:09:18 id=20085 trace_id=3 func=resolve_ip_tuple_fast line=5808 msg="Find an existing session, id-00022a0a, original direction"
2022-10-19 14:09:18 id=20085 trace_id=3 func=ipd_post_route_handler line=490 msg="out port2 vwl_zone_id 0, state2 0x0, quality 0.
"
2022-10-19 14:09:38 id=20085 trace_id=4 func=print_pkt_detail line=5727 msg="vd-root:0 received a packet(proto=17, 100.122.10.72:500-><HUB IP>:500) from local. "
2022-10-19 14:09:38 id=20085 trace_id=4 func=resolve_ip_tuple_fast line=5808 msg="Find an existing session, id-00022a0a, original direction"
2022-10-19 14:09:38 id=20085 trace_id=4 func=ipd_post_route_handler line=490 msg="out port2 vwl_zone_id 0, state2 0x0, quality 0.
"
2022-10-19 14:09:58 id=20085 trace_id=5 func=print_pkt_detail line=5727 msg="vd-root:0 received a packet(proto=17, 100.122.10.72:500-><HUB IP>:500) from local. "
2022-10-19 14:09:58 id=20085 trace_id=5 func=resolve_ip_tuple_fast line=5808 msg="Find an existing session, id-00022a0a, original direction"
2022-10-19 14:09:58 id=20085 trace_id=5 func=ipd_post_route_handler line=490 msg="out port2 vwl_zone_id 0, state2 0x0, quality 0.
"
2022-10-19 14:10:18 id=20085 trace_id=6 func=print_pkt_detail line=5727 msg="vd-root:0 received a packet(proto=17, 100.122.10.72:500-><HUB IP>:500) from local. "
2022-10-19 14:10:18 id=20085 trace_id=6 func=resolve_ip_tuple_fast line=5808 msg="Find an existing session, id-00022a0a, original direction"
2022-10-19 14:10:18 id=20085 trace_id=6 func=ipd_post_route_handler line=490 msg="out port2 vwl_zone_id 0, state2 0x0, quality 0.

 

==============================================

==============================================

 

HUB SIDE:

vd: root/0
name: Dial-Up_0
version: 1
interface: wan 5
addr: <HUB IP>:500 -> 145.224.100.83:1046
tun_id: 145.224.100.83/::10.0.0.31
remote_location: 0.0.0.0
virtual-interface-addr: 10.62.10.2 -> 0.0.0.0
created: 1411s ago
IKE SA: created 1/1 established 1/1 time 230/230/230 ms
IPsec SA: created 1/1 established 1/1 time 100/100/100 ms

id/spi: 19971 905126ead586fa65/c8fedf75665575f7
direction: responder
status: established 1411-1411s ago = 230ms
proposal: des-md5
key: 50fb3d8e88d8595f
lifetime/rekey: 86400/84718
DPD sent/recv: 00000000/000000d1

name=Dial-Up_0 ver=1 serial=27 <HUB IP>:0->145.224.100.83:0 tun_id=145.224.100.83 tun_id6=::10.0.0.31 dst_mtu=1500 dpd-link=on weight=1
bound_if=5 lgwy=static/1 tun=tunnel/255 mode=dial_inst/3 encap=none/8840 options[2288]=npu rgwy-chg frag-rfc run_state=0 accept_traffic=1 overlay_id=0
parent=Dial-Up index=0
proxyid_num=1 child_num=0 refcnt=5 ilast=19 olast=19 ad=/0
stat: rxp=0 txp=5 rxb=0 txb=420
dpd: mode=on-idle on=1 idle=60000ms retry=3 count=0 seqno=1
natt: mode=none draft=0 interval=0 remote_port=0
proxyid=Dial-Up proto=0 sa=1 ref=2 serial=1 add-route
src: 0:0.0.0.0/0.0.0.0:0
dst: 0:0.0.0.0/0.0.0.0:0
SA: ref=5 options=2a6 type=00 soft=0 mtu=1446 expire=41710/0B replaywin=2048
seqno=6 esn=0 replaywin_lastseq=00000000 itn=0 qat=0 hash_search_len=1
life: type=01 bytes=0/0 timeout=43186/43200
dec: spi=75712a5b esp=des key=8 f8d14d3a90643f03
ah=md5 key=16 527a434c60ac1b0e0a947c767e4cafc1
enc: spi=3ee3ff8d esp=des key=8 3a54253a29887249
ah=md5 key=16 14f51ad99e4c8afbb202db6e2ed5e68b
dec:pkts/bytes=0/0, enc:pkts/bytes=10/1100
npu_flag=01 npu_rgwy=145.224.100.83 npu_lgwy=<HUB IP> npu_selid=21 dec_npuid=0 enc_npuid=1


FGT # diagnose debug reset

FGT # dia debug flow filter saddr <HUB IP>

FGT # dia deb flow filter daddr 145.224.100.83

FGT # dia deb flow filter
vf: any
proto: any
Host addr: any
host saddr: <HUB IP>-<HUB IP>
host daddr: 145.224.100.83-145.224.100.83
port: any
sport: any
dport: any

FGT # dia deb flow show function-name enable
show function name

FGT # dia debug flow show iprope enable
show trace messages about iprope

FGT # dia debug console timestamp enable

FGT # dia debug flow trace start 999

FGT # dia debug enable

FGT # 2022-10-19 14:29:41 id=20085 trace_id=1 func=print_pkt_detail line=5863 msg="vd-root:0 received a packet(proto=17, <HUB IP>:500->145.224.100.83:1046) tun_id=0.0.0.0 from local. "
2022-10-19 14:29:41 id=20085 trace_id=1 func=resolve_ip_tuple_fast line=5949 msg="Find an existing session, id-013e953f, reply direction"
2022-10-19 14:30:01 id=20085 trace_id=2 func=print_pkt_detail line=5863 msg="vd-root:0 received a packet(proto=17, <HUB IP>:500->145.224.100.83:1046) tun_id=0.0.0.0 from local. "
2022-10-19 14:30:01 id=20085 trace_id=2 func=resolve_ip_tuple_fast line=5949 msg="Find an existing session, id-013e953f, reply direction"
2022-10-19 14:30:22 id=20085 trace_id=3 func=print_pkt_detail line=5863 msg="vd-root:0 received a packet(proto=17, <HUB IP>:500->145.224.100.83:1046) tun_id=0.0.0.0 from local. "
2022-10-19 14:30:22 id=20085 trace_id=3 func=resolve_ip_tuple_fast line=5949 msg="Find an existing session, id-013e953f, reply direction"
2022-10-19 14:31:02 id=20085 trace_id=4 func=print_pkt_detail line=5863 msg="vd-root:0 received a packet(proto=17, <HUB IP>:500->145.224.100.83:1046) tun_id=0.0.0.0 from local. "
2022-10-19 14:31:02 id=20085 trace_id=4 func=resolve_ip_tuple_fast line=5949 msg="Find an existing session, id-013e953f, reply direction"
2022-10-19 14:31:22 id=20085 trace_id=5 func=print_pkt_detail line=5863 msg="vd-root:0 received a packet(proto=17, <HUB IP>:500->145.224.100.83:1046) tun_id=0.0.0.0 from local. "
2022-10-19 14:31:22 id=20085 trace_id=5 func=resolve_ip_tuple_fast line=5949 msg="Find an existing session, id-013e953f, reply direction"
2022-10-19 14:31:42 id=20085 trace_id=6 func=print_pkt_detail line=5863 msg="vd-root:0 received a packet(proto=17, <HUB IP>:500->145.224.100.83:1046) tun_id=0.0.0.0 from local. "
2022-10-19 14:31:42 id=20085 trace_id=6 func=resolve_ip_tuple_fast line=5949 msg="Find an existing session, id-013e953f, reply direction"
2022-10-19 14:32:02 id=20085 trace_id=7 func=print_pkt_detail line=5863 msg="vd-root:0 received a packet(proto=17, <HUB IP>:500->145.224.100.83:1046) tun_id=0.0.0.0 from local. "
2022-10-19 14:32:02 id=20085 trace_id=7 func=resolve_ip_tuple_fast line=5949 msg="Find an existing session, id-013e953f, reply direction"