Description
This article describes that On-fabric endpoints connected to the EMS are not able to reach the configured ZTNA destinations. However, it works as expected when the same device is Off-fabric.
Scope
FortiGate, FortiClient EMS, ZTNA.
Solution
For Example:
On-fabric endpoint having IP address 10.0.0.201/24 trying to connect to a server over HTTPS 443 using ZTNA destination.
The server webpage does not open and gives the following error on the browser:
The configured proxy policy is as follows to allow the traffic to the ZTNA server:
There could be 2 issues:
Issue 1: Hair-pin NAT on traffic that is matching the ZTNA destinations.
FortiGate sniffer and debugs reveal that the traffic from the on-fabric endpoint is coming to the access-proxy gateway IP through the internal interface 'internal2' rather than on the external interface 'wan1”'
Therefore, dropping it due to 'msg="iprope_in_check() check failed on policy 0, drop”'.
2023-11-24 13:55:52.611254 internal2 in 10.0.0.201.59979 -> 172.17.97.169.3443: syn 2982667464
2023-11-24 13:55:52 id=65308 trace_id=103 func=print_pkt_detail line=5802 msg="vd-root:0 received a packet(proto=6, 10.0.0.201:59979->172.17.97.169:3443) tun_id=0.0.0.0 from internal2. flag [S], seq 2982667464, ack 0, win 64240"
2023-11-24 13:55:52 id=65308 trace_id=103 func=init_ip_session_common line=5980 msg="allocate a new session-0276c62f, tun_id=0.0.0.0"
2023-11-24 13:55:52 id=65308 trace_id=103 func=vf_ip_route_input_common line=2612 msg="find a route: flag=80000000 gw-172.17.97.169 via root"
2023-11-24 13:55:52 id=65308 trace_id=103 func=__iprope_tree_check line=529 msg="gnum-100004, use int hash, slot=86, len=7"
2023-11-24 13:55:52 id=65308 trace_id=103 func=get_new_addr line=1235 msg="find SNAT: IP-172.17.97.169(from IPPOOL), port-59979"
2023-11-24 13:55:52 id=65308 trace_id=103 func=fw_local_in_handler line=606 msg="iprope_in_check() check failed on policy 0, drop"
Solution:
Ideally, the on-fabric devices should have the ZTNA destination disabled so that it does not cause the issue (hair-pin NAT) as described above.
The ideal solution to this issue is shown in the following KB article:
However, in some cases, such as FortiSASE with ZTNA, it is not possible to create an On-fabric detection rule, it is possible to use the following workaround:
- Add the source-interface of the traffic in the source-interface field of the proxy policy. In this case, adding the internal2 as source-interface in the policy would resolve the issue as shown below:
2023-11-24 14:17:43.736421 internal2 in 10.0.0.201.60379 -> 172.17.97.169.3443: syn 965118737
2023-11-24 14:17:43.736653 internal2 out 172.17.97.169.3443 -> 10.0.0.201.60379: syn 1514755107 ack 965118738
2023-11-24 14:17:43.736813 internal2 in 10.0.0.201.60379 -> 172.17.97.169.3443: ack 1514755108
2023-11-24 14:17:43.736915 internal2 in 10.0.0.201.60379 -> 172.17.97.169.3443: psh 965118738 ack 1514755108
2023-11-24 14:17:43.736986 internal2 out 172.17.97.169.3443 -> 10.0.0.201.60379: ack 965118927
Issue 2: If FQDN is used for Destination Host, any traffic destined to that FQDN that does not match the ZTNA Connection Rule may fail.
Related document:
FQDN-based ZTNA TCP forwarding services
When FQDN-based ZTNA rules are created, FortiClient resolves each FQDN to a specific IP address such as 10.235.0.x (This is hard-coded and cannot be changed). Whenever a user makes a request for that FQDN, the FortiClient checks if there is any ZTNA destination rule for it or not.
If there is, then it intercepts the traffic and sends the traffic to the access-proxy. If there is not, then the traffic is just sent to the local default gateway and then dropped.
If the on-fabric endpoint needs to access the internal servers using FQDN and if there is no ZTNA destination rule for it, then the endpoint may not reach the internal servers.
For example:
Below is an on-fabric endpoint with FQDN-based ZTNA destinations configured:
The user tries to ping server.colombas.lab, the resolved IP comes as 10.235.0.2 whereas the real server IP is 10.22.0.200.
When the endpoint tries to reach server.colombas.lab:80, since there is no matching ZTNA rule, the endpoint will not be able to reach it.
Solution:
As mentioned above, ideally, the on-fabric devices should have the ZTNA Destination profile disabled so that the FortiClient does not change the IP of the FQDN.
The ideal solution to this issue is shown in the following document and KB article Link.
However, in some cases, such as FortiSASE with ZTNA, it is not possible to create ON-fabric detection rules, the following workaround can be used:
- Create a ZTNA destination rule to match with the FQDN: port that the endpoint is trying to reach. This is shown below:
Now since the endpoint has the matching ZTNA destination rule, the traffic is intercepted by the FortiClient and passed to the access-proxy and the destination is reachable.
When an endpoint is on-fabric with FQDN-based ZTNA destinations configured, the destination server FQDN and port must match one of the ZTNA destination rules. As expected, this will not work if the servers are accessed over non-TCP ports.
Note:
The above-mentioned workarounds are applicable to FortiClient v7.0.x. However, if the FortiClient is v7.2.0 and above, there is another workaround available as described below:
In FortiClient v7.2.0+, there is an option to disable ZTNA Destination for TCP Forwarding.
Navigate on FortiClient to ZTNA Destination -> Disable ZTNA destination.
Use Cases of this workaround:
- Administrators who do not have an option to create On-Fabric detection rules on EMS, such as while using FortiSASE (However, FortiSASE does not support FortiClient v7.2.0+).
- On-Fabric devices with ZTNA destinations enabled need to access the internal servers using non-TCP ports.
- Access to web servers protected via Access proxy via HTTP/HTTPS Access proxy type. Disabling this option from FortiClient does not disable the ZTNA Destination profile, hence its ZTNA certificate is not revoked.
