Remote Internet Access issue

[tedew]

Hello,

I have issue with RIA by Tunnel in case of main WAN link is down or IP SLA is not meet...

I have two sites  (MAIN and SMALL) and FGTs 100F in each. 

This SMALL Site has three interfaces:

WAN1 - Internet access

IPSec Tunnel0 - P2P Link between sites 

IPSec Tunnel1 - use WAN1 Internet Interface  

I have one SDWAN zone with all these interfaces above and two rule:

1. To_MAIN Site (use Tunnel0 -prefered,  and Tunnel1) (for traffic between sites)

2.To_Internet (use WAN - prefered, and Tunnel0 when WAN int has issue or is down (IP SLA))  

I have FW rule to from SMALL to MAIN site and destination interface is SDWAN zone

Issue is when WAN link has issue (or is down) and is removed from RoutingTable (thanks to SLA) -  Internet don't work, I see in debug that trafic is going to tunnel0 but not work, I found also in debug that FGT is still do  NAT using IP of WAN Interface , but why ?? When I disable NAT on FW rulle is working, but i need NAT when WAN link(Internet)  will be up again... 

LOG below:

2024-08-30 14:37:12 id=20085 trace_id=188 func=print_pkt_detail line=5867 msg="vd-root:0 received a packet(proto=1, 10.40.3.10:1->8.8.4.4:2048) tun_id=0.0.0.0 from vlan40. type=8, code=0, id=1, seq=2696."
2024-08-30 14:37:12 id=20085 trace_id=188 func=resolve_ip_tuple_fast line=5953 msg="Find an existing session, id-2d3ee778, original direction"
2024-08-30 14:37:12 id=20085 trace_id=188 func=npu_handle_session44 line=1183 msg="Trying to offloading session from vlan40 to Tunnel0-ToMain, skb.npu_flag=00000400 ses.state=00012204 ses.npu_state=0x05041008"
2024-08-30 14:37:12 id=20085 trace_id=188 func=fw_forward_dirty_handler line=410 msg="state=00012204, state2=00004001, npu_state=05041008"
2024-08-30 14:37:12 id=20085 trace_id=188 func=ids_receive line=418 msg="send to ips"
2024-08-30 14:37:12 id=20085 trace_id=188 func=__ip_session_run_tuple line=3487 msg="SNAT 10.40.3.10->169.xx.xx.10:60417"
2024-08-30 14:37:12 id=20085 trace_id=188 func=ipsecdev_hard_start_xmit line=669 msg="enter IPSec interface Tunnel0-ToMain, tun_id=0.0.0.0"
2024-08-30 14:37:12 id=20085 trace_id=188 func=_do_ipsecdev_hard_start_xmit line=229 msg="output to IPSec tunnel Tunnel0-ToMain"
2024-08-30 14:37:12 id=20085 trace_id=188 func=esp_output4 line=844 msg="IPsec encrypt/auth"
2024-08-30 14:37:12 id=20085 trace_id=188 func=ipsec_output_finish line=544 msg="send to 10.254.254.1 via intf-internal3"

Shouldn't be used for NAT IP of the Tunnel0 Interface ??

Thanks

[saleha]

Hello tedew,

Thank you for reaching out. If the route as you mentioned was already removed from routing table using the option to "update-static-route" in performance SLA I assume, this issue should not happen however I am suspecting the firewall still finding the route somehow and using the old route snat. Have you tried to enable "snat-route-change" as a fix:

Config sys global

set snat-route-change enable

end

Also make sure the firewall policy matching the traffic is not using an ip pool with external ip being the wan ip please.

Thank you,

saleha

[tedew]

Hello,

 I have "Use Outgoing Interface Address" in my rules. 

I have checked and  I don't have this option set ->  set snat-route-change enable

I read this post 
https://community.fortinet.com/t5/FortiGate/Technical-Tip-Implement-Source-NAT-for-IPsec-interface/t...

And i don't have set IP in IPSec on both sides, this also could be a reason ??

config system interface
edit "Tunnel0-ToMain"
set vdom "root"
set type tunnel
set snmp-index 20
set interface "internal6"
next

Setting IP on IPSec Interface can do some impact on current sessions between sites??

Thanks

[saleha]

Hi,

You can try setting the ipsec interface with a static ip yes I assumed you already had that enabled to use it for performance SLA. The option "snat-route-change" should still be available under the global config menu. you can try to locate it with the following command:
show full sys global | grep snat

Thank you,

saleha

[tedew]

Hello,
I have checked, and is disabled 
show full sys global | grep snat
set snat-route-change disable

[tedew]

Hello,

@saleha . so as action plan I should.

1.set snat-route-change enable

2.Set IP on both IPSec tunnels

correct ??

During setting these options i will break some connections ??

Thanks

[saleha]

yes you should plan a down time. I assume that the phase2 selectors on the ipsec tunnel at least on the direction from branch to main office is setup with default address - 0.0.0.0/0 - since it is for an RIA deployment. This is important because if you source nat the traffic the remote end of the tunnel should be able to allow this traffic based on natted address. That means the ipsec interface address should be allowed by a firewall policy as a source subnet on the remote/main office fortigate. You can test that with debug flow commands that you used to find out before to illustrate the issue:

di de flow filter addr 8.8.4.4

di de flow filter proto 1 <---- icmp protocol number

di de flow show function enable

di de flow trace strart 10

di de console time en

di de en

Thank you,

saleha

2- Firew

[Therejorty]

I had a similar issue before. When the WAN link drops, the FortiGate might still try to use the old NAT settings tied to that interface, causing traffic to get stuck. I fixed it by setting up a custom NAT policy that activates only when the WAN is down. This way, the device uses the correct NAT with Tunnel0. 

[tedew]

Hello,

@Therejorty , could You please provide more details how this custom NAT policy looks like :) ??

Thanks

[tedew]

But I have my FGT set like this, so I think no possible to set Custom NAT...