Problems with SSL certificate negotiation going through Fortigate

[fmmartins83]

I have a fortigate in the cloud that when the flow from a linux server passes through it, the source machine in question is on the Oracle Cloud internal network and the destination private load balancer is on another Oracle Cloud network, but I want to force them to communicate only by passing through the firewall. I can close telnet, which shows that the route is correct. I can connect to the destination, but when I try to execute the connection via the linux command line using SSL, it does not negotiate, giving an SSL Handshake error. The problem only occurs when passing through fortigate.

If I go through the VPN in others to the same Load Balancer it works normally, only when the connection is coming from an internal network to another internal network that the error occurs. The tests performed were done both with NAT active and without NAT active, but in both tests the error persisted.

[fmmartins83]

Diagnose debug enable without nat enabled:

fg-oci-hub-saopaulo # 2025-01-20 10:08:59 id=65308 trace_id=994 func=print_pkt_detail line=5932 msg="vd-root:0 received a packet(proto=6, 10.207.135.88:41074->10.204.5
.14:443) tun_id=0.0.0.0 from port2. flag [S], seq 1069083594, ack 0, win 35840"
2025-01-20 10:08:59 id=65308 trace_id=994 func=init_ip_session_common line=6124 msg="allocate a new session-001db062"
2025-01-20 10:08:59 id=65308 trace_id=994 func=iprope_dnat_check line=5480 msg="in-[port2], out-[]"
2025-01-20 10:08:59 id=65308 trace_id=994 func=iprope_dnat_tree_check line=824 msg="len=0"
2025-01-20 10:08:59 id=65308 trace_id=994 func=iprope_dnat_check line=5505 msg="result: skb_flags-02000000, vid-0, ret-no-match, act-accept, flag-00000000"
2025-01-20 10:08:59 id=65308 trace_id=994 func=__vf_ip_route_input_rcu line=1989 msg="find a route: flag=00000000 gw-10.200.1.17 via port2"
2025-01-20 10:08:59 id=65308 trace_id=994 func=__iprope_fwd_check line=810 msg="in-[port2], out-[port2], skb_flags-02000000, vid-0, app_id: 0, url_cat_id: 0"
2025-01-20 10:08:59 id=65308 trace_id=994 func=__iprope_tree_check line=524 msg="gnum-100004, use int hash, slot=36, len=2"
2025-01-20 10:08:59 id=65308 trace_id=994 func=__iprope_check_one_policy line=2140 msg="checked gnum-100004 policy-4, ret-matched, act-accept"
2025-01-20 10:08:59 id=65308 trace_id=994 func=__iprope_user_identity_check line=1903 msg="ret-matched"
2025-01-20 10:08:59 id=65308 trace_id=994 func=__iprope_check line=2404 msg="gnum-4e20, check-ffffffffa002cac7"
2025-01-20 10:08:59 id=65308 trace_id=994 func=__iprope_check_one_policy line=2140 msg="checked gnum-4e20 policy-6, ret-no-match, act-accept"
2025-01-20 10:08:59 id=65308 trace_id=994 func=__iprope_check_one_policy line=2140 msg="checked gnum-4e20 policy-6, ret-no-match, act-accept"
2025-01-20 10:08:59 id=65308 trace_id=994 func=__iprope_check_one_policy line=2140 msg="checked gnum-4e20 policy-6, ret-no-match, act-accept"
2025-01-20 10:08:59 id=65308 trace_id=994 func=__iprope_check line=2421 msg="gnum-4e20 check result: ret-no-match, act-accept, flag-00000000, flag2-00000000"
2025-01-20 10:08:59 id=65308 trace_id=994 func=__iprope_check_one_policy line=2374 msg="policy-4 is matched, act-accept"
2025-01-20 10:08:59 id=65308 trace_id=994 func=__iprope_fwd_check line=847 msg="after iprope_captive_check(): is_captive-0, ret-matched, act-accept, idx-4"
2025-01-20 10:08:59 id=65308 trace_id=994 func=iprope_fwd_auth_check line=876 msg="after iprope_captive_check(): is_captive-0, ret-matched, act-accept, idx-4"
2025-01-20 10:08:59 id=65308 trace_id=994 func=fw_forward_handler line=1001 msg="Allowed by Policy-4:"
2025-01-20 10:08:59 id=65308 trace_id=994 func=ip_session_confirm_final line=3131 msg="npu_state=0x101, hook=4"
2025-01-20 10:08:59 id=65308 trace_id=995 func=print_pkt_detail line=5932 msg="vd-root:0 received a packet(proto=6, 10.207.135.88:41074->10.204.5.14:443) tun_id=0.0.0.
0 from port2. flag [.], seq 1069083595, ack 3435938089, win 70"
2025-01-20 10:08:59 id=65308 trace_id=995 func=resolve_ip_tuple_fast line=6027 msg="Find an existing session, id-001db062, original direction"
2025-01-20 10:08:59 id=65308 trace_id=995 func=npu_handle_session44 line=1342 msg="Trying to offloading session from port2 to port2, skb.npu_flag=00000400 ses.state=00
000204 ses.npu_state=0x00000101"
2025-01-20 10:08:59 id=65308 trace_id=995 func=fw_forward_dirty_handler line=444 msg="state=00000204, state2=00000001, npu_state=00000101"
2025-01-20 10:08:59 id=65308 trace_id=996 func=print_pkt_detail line=5932 msg="vd-root:0 received a packet(proto=6, 10.207.135.88:41074->10.204.5.14:443) tun_id=0.0.0.
0 from port2. flag [.], seq 1069083595, ack 3435938089, win 70"
2025-01-20 10:08:59 id=65308 trace_id=996 func=resolve_ip_tuple_fast line=6027 msg="Find an existing session, id-001db062, original direction"
2025-01-20 10:08:59 id=65308 trace_id=996 func=npu_handle_session44 line=1342 msg="Trying to offloading session from port2 to port2, skb.npu_flag=00000400 ses.state=00
000204 ses.npu_state=0x00000101"
2025-01-20 10:08:59 id=65308 trace_id=996 func=fw_forward_dirty_handler line=444 msg="state=00000204, state2=00000001, npu_state=00000101"
2025-01-20 10:08:59 id=65308 trace_id=997 func=print_pkt_detail line=5932 msg="vd-root:0 received a packet(proto=6, 10.207.135.88:41074->10.204.5.14:443) tun_id=0.0.0.
0 from port2. flag [.], seq 1069084112, ack 3435938089, win 70"
2025-01-20 10:08:59 id=65308 trace_id=997 func=resolve_ip_tuple_fast line=6027 msg="Find an existing session, id-001db062, original direction"
2025-01-20 10:08:59 id=65308 trace_id=997 func=ipv4_fast_cb line=53 msg="enter fast path"
2025-01-20 10:09:59 id=65308 trace_id=998 func=print_pkt_detail line=5932 msg="vd-root:0 received a packet(proto=6, 10.207.135.88:41074->10.204.5.14:443) tun_id=0.0.0.
0 from port2. flag [.], seq 1069084112, ack 3435938089, win 70"
2025-01-20 10:09:59 id=65308 trace_id=998 func=resolve_ip_tuple_fast line=6027 msg="Find an existing session, id-001db062, original direction"
2025-01-20 10:09:59 id=65308 trace_id=998 func=ipv4_fast_cb line=53 msg="enter fast path"
2025-01-20 10:11:00 id=65308 trace_id=999 func=print_pkt_detail line=5932 msg="vd-root:0 received a packet(proto=6, 10.207.135.88:41074->10.204.5.14:443) tun_id=0.0.0.
0 from port2. flag [.], seq 1069084111, ack 3435938089, win 70"
2025-01-20 10:11:00 id=65308 trace_id=999 func=resolve_ip_tuple_fast line=6027 msg="Find an existing session, id-001db062, original direction"
2025-01-20 10:11:00 id=65308 trace_id=999 func=ipv4_fast_cb line=53 msg="enter fast path"

[fmmartins83 · Accepted solution]

Identified error in network interface MTU size. Adjusted to 9000 and fixed the error.