IPSEC tunnels behind CGNAT Starlink

[opt]

Hi all. We have one very interesting case. We using Fortigate HA routers on HQ and Branch.

Branch is connected to HQ via 2 providers over IPSEC-SD-WAN tunnels.

But now we have often problems with these 2 providers availibility and decided to try Starlink.

We have connected Starlink router to Fortigate, switched Starlink router to bypas mode. 

Now Branch's Fortigate behind Starlink's CGNAT with IP 100.122.N.N 255.192.0.0 and we can't connect classic peer-to-peer IPSEC as before with those 2 providers with public ip on both sides.

So the question is how to make connection between HQ and branches?

We tried configure IPSEC with dilaup user on HQ side as listener and remote side connect to HQ public ip.

The tunnel become UP but there is no traffic between routers. I sugges that there is some configuration mistakes, but need more experience to debug it.

[opt]

we swapped on hub side saddr and daddr and have got additional logs with iprope 

FGT # 2022-10-19 14:51:26 id=20085 trace_id=59 func=print_pkt_detail line=5863 msg="vd-root:0 received a packet(proto=17, 145.224.100.83:1046-><HUB IP>:500) tun_id=0.0.0.0 from wan. "
2022-10-19 14:51:26 id=20085 trace_id=59 func=resolve_ip_tuple_fast line=5949 msg="Find an existing session, id-013e953f, original direction"
2022-10-19 14:51:47 id=20085 trace_id=60 func=print_pkt_detail line=5863 msg="vd-root:0 received a packet(proto=17, 145.224.100.83:1046-><HUB IP>:500) tun_id=0.0.0.0 from wan. "
2022-10-19 14:51:47 id=20085 trace_id=60 func=resolve_ip_tuple_fast line=5949 msg="Find an existing session, id-013e953f, original direction"
2022-10-19 14:52:07 id=20085 trace_id=61 func=print_pkt_detail line=5863 msg="vd-root:0 received a packet(proto=17, 145.224.100.83:1046-><HUB IP>:500) tun_id=0.0.0.0 from wan. "
2022-10-19 14:52:07 id=20085 trace_id=61 func=resolve_ip_tuple_fast line=5949 msg="Find an existing session, id-013e953f, original direction"
2022-10-19 14:52:27 id=20085 trace_id=62 func=print_pkt_detail line=5863 msg="vd-root:0 received a packet(proto=17, 145.224.100.83:1046-><HUB IP>:500) tun_id=0.0.0.0 from wan. "
2022-10-19 14:52:27 id=20085 trace_id=62 func=resolve_ip_tuple_fast line=5949 msg="Find an existing session, id-013e953f, original direction"
2022-10-19 14:52:47 id=20085 trace_id=63 func=print_pkt_detail line=5863 msg="vd-root:0 received a packet(proto=17, 145.224.100.83:1046-><HUB IP>:500) tun_id=0.0.0.0 from wan. "
2022-10-19 14:52:47 id=20085 trace_id=63 func=resolve_ip_tuple_fast line=5949 msg="Find an existing session, id-013e953f, original direction"
2022-10-19 14:53:07 id=20085 trace_id=64 func=print_pkt_detail line=5863 msg="vd-root:0 received a packet(proto=17, 145.224.100.83:1046-><HUB IP>:500) tun_id=0.0.0.0 from wan. "
2022-10-19 14:53:07 id=20085 trace_id=64 func=resolve_ip_tuple_fast line=5949 msg="Find an existing session, id-013e953f, original direction"
2022-10-19 14:53:07 id=20085 trace_id=64 func=vf_ip_route_input_common line=2605 msg="find a route: flag=80000000 gw-<HUB IP> via root"
2022-10-19 14:53:07 id=20085 trace_id=64 func=iprope_access_proxy_check line=436 msg="in-[wan], out-[], skb_flags-02000100, vid-0"
2022-10-19 14:53:07 id=20085 trace_id=64 func=__iprope_check line=2276 msg="gnum-100017, check-ffffffbffc02b040"
2022-10-19 14:53:07 id=20085 trace_id=64 func=iprope_policy_group_check line=4734 msg="after check: ret-no-match, act-accept, flag-00000000, flag2-00000000"
2022-10-19 14:53:07 id=20085 trace_id=64 func=iprope_in_check line=469 msg="in-[wan], out-[], skb_flags-02000100, vid-0"
2022-10-19 14:53:07 id=20085 trace_id=64 func=__iprope_check line=2276 msg="gnum-100011, check-ffffffbffc02bfe0"
2022-10-19 14:53:07 id=20085 trace_id=64 func=iprope_policy_group_check line=4734 msg="after check: ret-no-match, act-drop, flag-00000000, flag2-00000000"
2022-10-19 14:53:07 id=20085 trace_id=64 func=__iprope_check line=2276 msg="gnum-100001, check-ffffffbffc02b040"
2022-10-19 14:53:07 id=20085 trace_id=64 func=iprope_policy_group_check line=4734 msg="after check: ret-no-match, act-accept, flag-00000000, flag2-00000000"
2022-10-19 14:53:07 id=20085 trace_id=64 func=__iprope_check line=2276 msg="gnum-10000e, check-ffffffbffc02b040"
2022-10-19 14:53:07 id=20085 trace_id=64 func=__iprope_check_one_policy line=2029 msg="checked gnum-10000e policy-4294967295, ret-no-match, act-accept"
2022-10-19 14:53:07 id=20085 trace_id=64 func=__iprope_check_one_policy line=2029 msg="checked gnum-10000e policy-4294967295, ret-no-match, act-accept"
2022-10-19 14:53:07 id=20085 trace_id=64 func=__iprope_check_one_policy line=2029 msg="checked gnum-10000e policy-4294967295, ret-matched, act-accept"
2022-10-19 14:53:07 id=20085 trace_id=64 func=__iprope_check_one_policy line=2246 msg="policy-4294967295 is matched, act-accept"
2022-10-19 14:53:07 id=20085 trace_id=64 func=__iprope_check line=2293 msg="gnum-10000e check result: ret-matched, act-accept, flag-00000000, flag2-00000000"
2022-10-19 14:53:07 id=20085 trace_id=64 func=iprope_policy_group_check line=4734 msg="after check: ret-matched, act-accept, flag-00000000, flag2-00000000"
2022-10-19 14:53:27 id=20085 trace_id=65 func=print_pkt_detail line=5863 msg="vd-root:0 received a packet(proto=17, 145.224.100.83:1046-><HUB IP>:500) tun_id=0.0.0.0 from wan. "
2022-10-19 14:53:27 id=20085 trace_id=65 func=resolve_ip_tuple_fast line=5949 msg="Find an existing session, id-013e953f, original direction"
2022-10-19 14:53:47 id=20085 trace_id=66 func=print_pkt_detail line=5863 msg="vd-root:0 received a packet(proto=17, 145.224.100.83:1046-><HUB IP>:500) tun_id=0.0.0.0 from wan. "
2022-10-19 14:53:47 id=20085 trace_id=66 func=resolve_ip_tuple_fast line=5949 msg="Find an existing session, id-013e953f, original direction"
2022-10-19 14:54:07 id=20085 trace_id=67 func=print_pkt_detail line=5863 msg="vd-root:0 received a packet(proto=17, 145.224.100.83:1046-><HUB IP>:500) tun_id=0.0.0.0 from wan. "
2022-10-19 14:54:07 id=20085 trace_id=67 func=resolve_ip_tuple_fast line=5949 msg="Find an existing session, id-013e953f, original direction"

[opt · Accepted solution]

The link from Starlink should be connected directly to FGT port. When link connected thru switch's vlan strange things happening - ICMP work but other traffic not flow.

Found some information on Starlink's support page:

Can I use a network switch with Starlink?

Yes, you are welcome to connect your own equipment to Starlink. However, we cannot guarantee Starlink performance or compatibility with third party networking devices.

The case is closed because now router behind Starlink is connecting as dialup ipsec client to Fortigate with NAT-T.

[scliff]

Did you get this sorted out? I am trying to do this behind some LTE carriers and am not quite sure I follow the threads.

[Toshi_Esumi]

Nothing should be special with either Starlink or 4G/LTE other than the public IP (or private IP) your device would pull is dynamic. With LTE you have an option to buy a static IP service from the carrier though, which Starlink doesn't offer.

So you have to set up either agressive mode (IKEv1) or dynamic (IKEv2) IPsec.

Toshi

[scliff]

I tried a dymanic DNS tunnel but it tries to connect back to the IP which is behind CGNAT. I am going to try to figure out how to do a Dial up , this would be ideal for my workstation and our remote offices that are not using static IP <for whatever reason they cannot :)>

Thanks

[Toshi_Esumi]

What do you mean by "DNS tunnel"? You have an internal DNS server at the end of IPSec tunnel?

Toshi

[scliff]

using dynamic DNS for the FortiGate to resolve to. I think I need to use dialup for cgnat but trying to figure that out.

[Toshi_Esumi]

I guess DDNS wouldn't work well over CGNAT. A regular aggressive mode should work fine. I think I tested IKEv2 dynamic when I tested with Starlink.

Toshi