Outbound FTP and FTPs connections result in Client-RST and never allow connection

[bcote]

Hi ,

since we migrated to a Fortigate solution, I have been having some issues with connecting to outbound FTP servers. Currently, I have an issue with 2 government entities that require us to upload information to their systems. One uses plain FTP(in this day and age...) and the other FTPs... nothing to help me out. So far the same result happens when I use a client or server that resides behind my Fortigate and tries to do an outbound connection. I have tried different FTP clients(Filezilla, CoreFTP, and others) with no positive results. When I look through the forward Traffic, I see the outbound connections being detected by the Application control applied on my policy, states that it is allowed and that the action is Client-RST. I've read that the RST isn't necessarily indication that something went wrong, but so far it is the only thing I can look into. I have opened a ticket with support, but in the meantime, I was wondering if any of you had experienced this type of issue beforehand.

I can use the same configurations(on the client FTP) on my laptop which uses a different line to bypass the firewall and I am able to connect and get a directory listing. When I am through the Fortigate, I get "connection successful" but then it hangs at directory listing and after 20 seconds, timeouts and retries. I've tried removing all Security policies, removing outbound firewall NAT to simply use the ISP provided IP.. nothing seems to matter. 

I expect FTPs to be a bit of an issue simply due to the nature of it, but FTP on port 21.. why would this be an issue? 

Any input would be appreciated.

Ben

[oheigl]

Did you modify something with the session helper regarding the FTP? Is it working if you set the outgoing policy to any/all service? It seems like the data channel on the random high port is not working, that's most likely a session helper problem.

Can you do a debug flow of the connection? (http://kb.fortinet.com/kb/viewContent.do?externalId=FD30038 - Step 4)

[bcote]

Hi Oheigl,

as far as I can tell, I don't have an FTP session-helper... The only one I did remove was the SIP-ALG as it was actually causing us issues with our SIP provider, but the other 19 are still all there but there isn't anything related to FTP. So I added the session-helper and rebooted the firewall and still no joy.

I did try to do a full Any/all policy initially thinking it was most likely a rule or Security services blocking, but it still wouldn't go through.

I am not as familiar with debug trace on fortigate but if I follow the steps in the KB you linked, I see many traces to my Gateway (SDWAN to public NAT IP), but it doesn't give me much detail on the FTP connection itself, nor do I see any type of deny, only a ton of allows.

here is what comes out(with modified information) if that can indicate something to someone.

id=20085 trace_id=1462 func=print_pkt_detail line=5363 msg="vd-root received a packet(proto=6, 10.1.99.243:50161->External FTPServer:21) from LAN-LACP-P39P40. flag , seq 1871012443, ack 0, win 64240" id=20085 trace_id=1462 func=init_ip_session_common line=5519 msg="allocate a new session-227bb1ba" id=20085 trace_id=1462 func=vf_ip_route_input_common line=2583 msg="find a route: flag=04000000 gw-GW-ISP-IP via port17" id=20085 trace_id=1462 func=fw_forward_handler line=737 msg="Allowed by Policy-1: SNAT" id=20085 trace_id=1462 func=__ip_session_run_tuple line=3164 msg="SNAT 10.1.99.243->GW-SNAT-IP:50161" id=20085 trace_id=1462 func=np6_hif_nturbo_build_vtag line=867 msg="np6_hif_nturbo_build_vtag: vtag->magic d153beef, vtag->coretag 138, vtag->vid 0 vtag->sip[0] 1f69a8e, vtag->sip[1] 0, vtag->sip[2] 0, vtag->sip[3] 0 vtag->sport 61891, vtag->mtu 1500, vtag->flags 12, vtag->np6_index 1" id=20085 trace_id=1463 func=print_pkt_detail line=5363 msg="vd-root received a packet(proto=6, External FTPServer:21->GW-SNAT-IP:50161) from port18. flag [S.], seq 526153893, ack 1871012444, win 29200" id=20085 trace_id=1463 func=resolve_ip_tuple_fast line=5438 msg="Find an existing session, id-227bb1ba, reply direction" id=20085 trace_id=1463 func=__ip_session_run_tuple line=3178 msg="DNAT GW-SNAT-IP:50161->10.1.99.243:50161" id=20085 trace_id=1463 func=vf_ip_route_input_common line=2583 msg="find a route: flag=00000000 gw-10.1.99.243 via LAN-LACP-P39P40" id=20085 trace_id=1463 func=np6_hif_nturbo_build_vtag line=867 msg="np6_hif_nturbo_build_vtag: vtag->magic d153beef, vtag->coretag 224, vtag->vid 0 vtag->sip[0] 0, vtag->sip[1] 0, vtag->sip[2] 0, vtag->sip[3] 0 vtag->sport 0, vtag->mtu 1500, vtag->flags 1, vtag->np6_index 69" id=20085 trace_id=1464 func=print_pkt_detail line=5363 msg="vd-root received a packet(proto=6, 10.1.99.243:50161->External FTPServer:21) from LAN-LACP-P39P40. flag [.], seq 1871012444, ack 526153894, win 257" id=20085 trace_id=1464 func=resolve_ip_tuple_fast line=5438 msg="Find an existing session, id-227bb1ba, original direction" id=20085 trace_id=1464 func=vf_ip_route_input_common line=2583 msg="find a route: flag=04000000 gw-GW-ISP-IP via port17" id=20085 trace_id=1464 func=__ip_session_run_tuple line=3164 msg="SNAT 10.1.99.243->GW-SNAT-IP:50161" id=20085 trace_id=1464 func=np6_hif_nturbo_build_vtag line=867 msg="np6_hif_nturbo_build_vtag: vtag->magic d153beef, vtag->coretag 138, vtag->vid 0 vtag->sip[0] 1f69a8e, vtag->sip[1] 0, vtag->sip[2] 0, vtag->sip[3] 0 vtag->sport 61891, vtag->mtu 1500, vtag->flags 2, vtag->np6_index 1" id=20085 trace_id=1465 func=print_pkt_detail line=5363 msg="vd-root received a packet(proto=6, External FTPServer:21->GW-SNAT-IP:50161) from port18. flag [.], seq 526153894, ack 1871012444, win 229" id=20085 trace_id=1465 func=resolve_ip_tuple_fast line=5438 msg="Find an existing session, id-227bb1ba, reply direction" id=20085 trace_id=1465 func=__ip_session_run_tuple line=3178 msg="DNAT GW-SNAT-IP:50161->10.1.99.243:50161" id=20085 trace_id=1465 func=vf_ip_route_input_common line=2583 msg="find a route: flag=00000000 gw-10.1.99.243 via LAN-LACP-P39P40" id=20085 trace_id=1465 func=np6_hif_nturbo_build_vtag line=867 msg="np6_hif_nturbo_build_vtag: vtag->magic d153beef, vtag->coretag 224, vtag->vid 0 vtag->sip[0] 0, vtag->sip[1] 0, vtag->sip[2] 0, vtag->sip[3] 0 vtag->sport 0, vtag->mtu 1500, vtag->flags 1, vtag->np6_index 69" id=20085 trace_id=1466 func=print_pkt_detail line=5363 msg="vd-root received a packet(proto=6, 10.1.99.243:50161->External FTPServer:21) from LAN-LACP-P39P40. flag [.], seq 1871012444, ack 526153914, win 257" id=20085 trace_id=1466 func=resolve_ip_tuple_fast line=5438 msg="Find an existing session, id-227bb1ba, original direction" id=20085 trace_id=1466 func=vf_ip_route_input_common line=2583 msg="find a route: flag=04000000 gw-GW-ISP-IP via port17" id=20085 trace_id=1466 func=__ip_session_run_tuple line=3164 msg="SNAT 10.1.99.243->GW-SNAT-IP:50161" id=20085 trace_id=1466 func=np6_hif_nturbo_build_vtag line=867 msg="np6_hif_nturbo_build_vtag: vtag->magic d153beef, vtag->coretag 138, vtag->vid 0 vtag->sip[0] 1f69a8e, vtag->sip[1] 0, vtag->sip[2] 0, vtag->sip[3] 0 vtag->sport 61891, vtag->mtu 1500, vtag->flags 2, vtag->np6_index 1" id=20085 trace_id=1467 func=print_pkt_detail line=5363 msg="vd-root received a packet(proto=6, External FTPServer:21->GW-SNAT-IP:50161) from port18. flag [.], seq 526153914, ack 1871012454, win 229" id=20085 trace_id=1467 func=resolve_ip_tuple_fast line=5438 msg="Find an existing session, id-227bb1ba, reply direction" id=20085 trace_id=1467 func=__ip_session_run_tuple line=3178 msg="DNAT GW-SNAT-IP:50161->10.1.99.243:50161" id=20085 trace_id=1467 func=vf_ip_route_input_common line=2583 msg="find a route: flag=00000000 gw-10.1.99.243 via LAN-LACP-P39P40" id=20085 trace_id=1467 func=np6_hif_nturbo_build_vtag line=867 msg="np6_hif_nturbo_build_vtag: vtag->magic d153beef, vtag->coretag 224, vtag->vid 0 vtag->sip[0] 0, vtag->sip[1] 0, vtag->sip[2] 0, vtag->sip[3] 0 vtag->sport 0, vtag->mtu 1500, vtag->flags 1, vtag->np6_index 69" id=20085 trace_id=1468 func=print_pkt_detail line=5363 msg="vd-root received a packet(proto=6, External FTPServer:21->GW-SNAT-IP:50161) from port18. flag [.], seq 526153914, ack 1871012454, win 229" id=20085 trace_id=1468 func=resolve_ip_tuple_fast line=5438 msg="Find an existing session, id-227bb1ba, reply direction" id=20085 trace_id=1468 func=__ip_session_run_tuple line=3178 msg="DNAT GW-SNAT-IP:50161->10.1.99.243:50161" id=20085 trace_id=1468 func=vf_ip_route_input_common line=2583 msg="find a route: flag=00000000 gw-10.1.99.243 via LAN-LACP-P39P40" id=20085 trace_id=1468 func=np6_hif_nturbo_build_vtag line=867 msg="np6_hif_nturbo_build_vtag: vtag->magic d153beef, vtag->coretag 224, vtag->vid 0 vtag->sip[0] 0, vtag->sip[1] 0, vtag->sip[2] 0, vtag->sip[3] 0 vtag->sport 0, vtag->mtu 1500, vtag->flags 1, vtag->np6_index 69" id=20085 trace_id=1469 func=print_pkt_detail line=5363 msg="vd-root received a packet(proto=6, 10.1.99.243:50161->External FTPServer:21) from LAN-LACP-P39P40. flag [.], seq 1871012454, ack 526153952, win 257" id=20085 trace_id=1469 func=resolve_ip_tuple_fast line=5438 msg="Find an existing session, id-227bb1ba, original direction" id=20085 trace_id=1469 func=vf_ip_route_input_common line=2583 msg="find a route: flag=04000000 gw-GW-ISP-IP via port17" id=20085 trace_id=1469 func=__ip_session_run_tuple line=3164 msg="SNAT 10.1.99.243->GW-SNAT-IP:50161" id=20085 trace_id=1469 func=np6_hif_nturbo_build_vtag line=867 msg="np6_hif_nturbo_build_vtag: vtag->magic d153beef, vtag->coretag 138, vtag->vid 0 vtag->sip[0] 1f69a8e, vtag->sip[1] 0, vtag->sip[2] 0, vtag->sip[3] 0 vtag->sport 61891, vtag->mtu 1500, vtag->flags 2, vtag->np6_index 1" id=20085 trace_id=1470 func=print_pkt_detail line=5363 msg="vd-root received a packet(proto=6, External FTPServer:21->GW-SNAT-IP:50161) from port18. flag [.], seq 526153952, ack 1871012464, win 229" id=20085 trace_id=1470 func=resolve_ip_tuple_fast line=5438 msg="Find an existing session, id-227bb1ba, reply direction" id=20085 trace_id=1470 func=__ip_session_run_tuple line=3178 msg="DNAT GW-SNAT-IP:50161->10.1.99.243:50161" id=20085 trace_id=1470 func=vf_ip_route_input_common line=2583 msg="find a route: flag=00000000 gw-10.1.99.243 via LAN-LACP-P39P40" id=20085 trace_id=1470 func=np6_hif_nturbo_build_vtag line=867 msg="np6_hif_nturbo_build_vtag: vtag->magic d153beef, vtag->coretag 224, vtag->vid 0 vtag->sip[0] 0, vtag->sip[1] 0, vtag->sip[2] 0, vtag->sip[3] 0 vtag->sport 0, vtag->mtu 1500, vtag->flags 1, vtag->np6_index 69" id=20085 trace_id=1471 func=print_pkt_detail line=5363 msg="vd-root received a packet(proto=6, 10.1.99.243:50161->External FTPServer:21) from LAN-LACP-P39P40. flag [.], seq 1871012464, ack 526153990, win 257" id=20085 trace_id=1471 func=resolve_ip_tuple_fast line=5438 msg="Find an existing session, id-227bb1ba, original direction" id=20085 trace_id=1471 func=vf_ip_route_input_common line=2583 msg="find a route: flag=04000000 gw-GW-ISP-IP via port17" id=20085 trace_id=1471 func=__ip_session_run_tuple line=3164 msg="SNAT 10.1.99.243->GW-SNAT-IP:50161" id=20085 trace_id=1471 func=np6_hif_nturbo_build_vtag line=867 msg="np6_hif_nturbo_build_vtag: vtag->magic d153beef, vtag->coretag 138, vtag->vid 0 vtag->sip[0] 1f69a8e, vtag->sip[1] 0, vtag->sip[2] 0, vtag->sip[3] 0 vtag->sport 61891, vtag->mtu 1500, vtag->flags 2, vtag->np6_index 1" id=20085 trace_id=1472 func=print_pkt_detail line=5363 msg="vd-root received a packet(proto=6, External FTPServer:21->GW-SNAT-IP:50161) from port18. flag [.], seq 526153990, ack 1871012478, win 229" id=20085 trace_id=1472 func=resolve_ip_tuple_fast line=5438 msg="Find an existing session, id-227bb1ba, reply direction" id=20085 trace_id=1472 func=__ip_session_run_tuple line=3178 msg="DNAT GW-SNAT-IP:50161->10.1.99.243:50161" id=20085 trace_id=1472 func=vf_ip_route_input_common line=2583 msg="find a route: flag=00000000 gw-10.1.99.243 via LAN-LACP-P39P40" id=20085 trace_id=1472 func=np6_hif_nturbo_build_vtag line=867 msg="np6_hif_nturbo_build_vtag: vtag->magic d153beef, vtag->coretag 224, vtag->vid 0 vtag->sip[0] 0, vtag->sip[1] 0, vtag->sip[2] 0, vtag->sip[3] 0 vtag->sport 0, vtag->mtu 1500, vtag->flags 1, vtag->np6_index 69" id=20085 trace_id=1473 func=print_pkt_detail line=5363 msg="vd-root received a packet(proto=6, 10.1.99.243:50161->External FTPServer:21) from LAN-LACP-P39P40. flag [.], seq 1871012478, ack 526154024, win 257" id=20085 trace_id=1473 func=resolve_ip_tuple_fast line=5438 msg="Find an existing session, id-227bb1ba, original direction" id=20085 trace_id=1473 func=vf_ip_route_input_common line=2583 msg="find a route: flag=04000000 gw-GW-ISP-IP via port17" id=20085 trace_id=1473 func=__ip_session_run_tuple line=3164 msg="SNAT 10.1.99.243->GW-SNAT-IP:50161" id=20085 trace_id=1473 func=np6_hif_nturbo_build_vtag line=867 msg="np6_hif_nturbo_build_vtag: vtag->magic d153beef, vtag->coretag 138, vtag->vid 0 vtag->sip[0] 1f69a8e, vtag->sip[1] 0, vtag->sip[2] 0, vtag->sip[3] 0 vtag->sport 61891, vtag->mtu 1500, vtag->flags 2, vtag->np6_index 1" id=20085 trace_id=1474 func=print_pkt_detail line=5363 msg="vd-root received a packet(proto=6, External FTPServer:21->GW-SNAT-IP:50161) from port18. flag [.], seq 526154024, ack 1871012491, win 229" id=20085 trace_id=1474 func=resolve_ip_tuple_fast line=5438 msg="Find an existing session, id-227bb1ba, reply direction" id=20085 trace_id=1474 func=__ip_session_run_tuple line=3178 msg="DNAT GW-SNAT-IP:50161->10.1.99.243:50161" id=20085 trace_id=1474 func=vf_ip_route_input_common line=2583 msg="find a route: flag=00000000 gw-10.1.99.243 via LAN-LACP-P39P40" id=20085 trace_id=1474 func=np6_hif_nturbo_build_vtag line=867 msg="np6_hif_nturbo_build_vtag: vtag->magic d153beef, vtag->coretag 224, vtag->vid 0 vtag->sip[0] 0, vtag->sip[1] 0, vtag->sip[2] 0, vtag->sip[3] 0 vtag->sport 0, vtag->mtu 1500, vtag->flags 1, vtag->np6_index 69"

[oheigl]

Regarding the session-helper, you can check it with the following command, I think the example is default configuration:

show system session-helper | grep -f ftp

config system session-helper
edit 5
set name tftp <---
set protocol 17
set port 69
next
edit 9
set name ftp <---
set protocol 6
set port 21
next
end

The debug flow looks good as far as the initial connection goes, but we already knew that. The problem is also that the ALL ALL ANY rule doesn't help, because the transport is going through another high port, which should be forwarded to the client similar to the destination nat on port 21.

Can you give me one other information, please do a sniffer trace like this:

diag sniffer packet any 'host <ftp_server_ip>' 4

After you started the trace, please try to connect again. If we can see in this output that the FTP server is trying to connect to the high port, but it's not forwarded to your LAN client, the session helper is not working.

[bcote]

Thanks for the quick response. Here is the output of the trace You can ignore the 10.15.1.246 as it is a different application connecting to that same FTP server(government server being used for more than just FTP).

To give you a little bit of configuration info :

Port 17-18 are 1Gb ISP links configured in SD-WAN 

Port 39-40 are 10Gb LACP LAN ports. 

Seeing as the packets seem to leave from one interface and come back the other, I wonder if the firewall doesn't want to complete the connection. The more I think about it, I think the date the users are telling me this stopped working might be the day we added the 2nd line. My configuration was always through an SD-WAN connection since I knew we were adding a 2nd connection this summer, but seeing this trace just made me realize that it could potentially be the issue. I am configured for 50% on each line base off of volume, but I didn't think an outgoing packet wouldn't automatically come back the same interface like it clearly shows right now.

I could easily change the algorithm right now to only go through 1 line and that could put this theory to rest, but I will wait for your input. Thanks

Primary-C3030 (root) # diag sniffer packet any 'host EX FTP Server' 4 interfaces=[any] filters=[host EX FTP Server] 7.433427 port39 in 10.15.1.246.49412 -> EX FTP Server.5800: psh 2126391509 ack 2027176957 7.433429 LAN-LACP-P39P40 in 10.15.1.246.49412 -> EX FTP Server.5800: psh 2126391509 ack 2027176957 7.433441 port17 out GW SNAT IP.49412 -> EX FTP Server.5800: psh 2126391509 ack 2027176957 7.443908 port18 in EX FTP Server.5800 -> GW SNAT IP.49412: ack 2126391512 7.443915 LAN-LACP-P39P40 out EX FTP Server.5800 -> 10.15.1.246.49412: ack 2126391512 7.443917 port40 out EX FTP Server.5800 -> 10.15.1.246.49412: ack 2126391512 15.543884 port40 in 10.1.99.243.58333 -> EX FTP Server.21: syn 2556059931 15.543886 LAN-LACP-P39P40 in 10.1.99.243.58333 -> EX FTP Server.21: syn 2556059931 15.544060 port17 out GW SNAT IP.58333 -> EX FTP Server.21: syn 2556059931 15.553683 port18 in EX FTP Server.21 -> GW SNAT IP.58333: syn 67220854 ack 2556059932 15.554169 LAN-LACP-P39P40 out EX FTP Server.21 -> 10.1.99.243.58333: syn 67220854 ack 2556059932 15.554171 port39 out EX FTP Server.21 -> 10.1.99.243.58333: syn 67220854 ack 2556059932 15.555715 port40 in 10.1.99.243.58333 -> EX FTP Server.21: ack 67220855 15.555717 LAN-LACP-P39P40 in 10.1.99.243.58333 -> EX FTP Server.21: ack 67220855 15.555746 port17 out GW SNAT IP.58333 -> EX FTP Server.21: ack 67220855 15.567298 port18 in EX FTP Server.21 -> GW SNAT IP.58333: psh 67220855 ack 2556059932 15.567358 LAN-LACP-P39P40 out EX FTP Server.21 -> 10.1.99.243.58333: psh 67220855 ack 2556059932 15.567361 port39 out EX FTP Server.21 -> 10.1.99.243.58333: psh 67220855 ack 2556059932 15.569068 port40 in 10.1.99.243.58333 -> EX FTP Server.21: psh 2556059932 ack 67220875 15.569070 LAN-LACP-P39P40 in 10.1.99.243.58333 -> EX FTP Server.21: psh 2556059932 ack 67220875 15.569206 port17 out GW SNAT IP.58333 -> EX FTP Server.21: psh 2556059932 ack 67220875 15.578624 port18 in EX FTP Server.21 -> GW SNAT IP.58333: ack 2556059942 15.578632 port18 in EX FTP Server.21 -> GW SNAT IP.58333: psh 67220875 ack 2556059942 15.578664 LAN-LACP-P39P40 out EX FTP Server.21 -> 10.1.99.243.58333: ack 2556059942 15.578666 port39 out EX FTP Server.21 -> 10.1.99.243.58333: ack 2556059942 15.578668 LAN-LACP-P39P40 out EX FTP Server.21 -> 10.1.99.243.58333: psh 67220875 ack 2556059942 15.578669 port39 out EX FTP Server.21 -> 10.1.99.243.58333: psh 67220875 ack 2556059942 15.580574 port40 in 10.1.99.243.58333 -> EX FTP Server.21: psh 2556059942 ack 67220913 15.580576 LAN-LACP-P39P40 in 10.1.99.243.58333 -> EX FTP Server.21: psh 2556059942 ack 67220913 15.580621 port17 out GW SNAT IP.58333 -> EX FTP Server.21: psh 2556059942 ack 67220913 15.590038 port18 in EX FTP Server.21 -> GW SNAT IP.58333: psh 67220913 ack 2556059952 15.590075 LAN-LACP-P39P40 out EX FTP Server.21 -> 10.1.99.243.58333: psh 67220913 ack 2556059952 15.590078 port39 out EX FTP Server.21 -> 10.1.99.243.58333: psh 67220913 ack 2556059952 15.591824 port40 in 10.1.99.243.58333 -> EX FTP Server.21: psh 2556059952 ack 67220951 15.591825 LAN-LACP-P39P40 in 10.1.99.243.58333 -> EX FTP Server.21: psh 2556059952 ack 67220951 15.591875 port17 out GW SNAT IP.58333 -> EX FTP Server.21: psh 2556059952 ack 67220951 15.601286 port18 in EX FTP Server.21 -> GW SNAT IP.58333: psh 67220951 ack 2556059966 15.601321 LAN-LACP-P39P40 out EX FTP Server.21 -> 10.1.99.243.58333: psh 67220951 ack 2556059966 15.601324 port39 out EX FTP Server.21 -> 10.1.99.243.58333: psh 67220951 ack 2556059966 15.603245 port40 in 10.1.99.243.58333 -> EX FTP Server.21: psh 2556059966 ack 67220985 15.603247 LAN-LACP-P39P40 in 10.1.99.243.58333 -> EX FTP Server.21: psh 2556059966 ack 67220985 15.603296 port17 out GW SNAT IP.58333 -> EX FTP Server.21: psh 2556059966 ack 67220985 15.652950 port18 in EX FTP Server.21 -> GW SNAT IP.58333: ack 2556059979 15.652968 LAN-LACP-P39P40 out EX FTP Server.21 -> 10.1.99.243.58333: ack 2556059979 15.652970 port39 out EX FTP Server.21 -> 10.1.99.243.58333: ack 2556059979 17.886790 port39 in 10.15.1.243.49282 -> EX FTP Server.5811: psh 2443566249 ack 754380447 17.886792 LAN-LACP-P39P40 in 10.15.1.243.49282 -> EX FTP Server.5811: psh 2443566249 ack 754380447 17.886811 port17 out GW SNAT IP.49282 -> EX FTP Server.5811: psh 2443566249 ack 754380447 17.909631 port18 in EX FTP Server.5811 -> GW SNAT IP.49282: psh 754380447 ack 2443566308 17.909641 LAN-LACP-P39P40 out EX FTP Server.5811 -> 10.15.1.243.49282: psh 754380447 ack 2443566308 17.909643 port40 out EX FTP Server.5811 -> 10.15.1.243.49282: psh 754380447 ack 2443566308 .... 

[oheigl]

Well that's interesting, also it's the same with the LAN side packets, sometimes it's port39 out and the reply comes through port40 in. Are the two WAN interfaces configured with different IP addresses and networks (completely different subnet with different gateways?)

I'm not sure how this is working, do you have asymmetric routing enabled on your FortiGate? Also is the FTP traffic just plain FTP or something like FTPS?

[bcote]

Yeah I noticed that too for the LAN ports. As far as the WAN interfaces, they are from the same provider, but going to different locations for redundancy. Different Subnet and Gateway completely.

It is any type of FTP. I'm trying FTP, FTPs, SFTP... But seeing this trace, I wouldn't be surprised if more was affected, I simply wasn't made aware of it. 

Asymmetric routing is not enabled, but clearly my routing is done in a way where it should be. I have a call with Fortinet this morning to address this so I'm sure we will be working on modifying some configurations at the routing level, which hopefully in turn fixes the connection issues. 

I will update this post after the call.

Thanks,

[oheigl]

That should be impossible then, how could that work if the packet is leaving the FortiGate with the NAT address e.g. from port17 and the reply is processed by the interface port18 with a completely different address (and still the session information is correct and forwarded by the firewall)

I'm really interested in the outcome of your call, thanks!

[bcote]

Hi oheigl,

so it is still ongoing as we ''fixed'' part of the issue, but FTPs remains impossible to connect. As far as FTP, I think you were spot on. As the engineer got familiar with my setup, he noticed that I was using an SD-WAN connection and using an IPPool NAT address to send the traffic out. After running many traces and confirming it wasn't a security policy or specific firewall rule, we simply created a temporary rule for all FTP traffic to go out the SD-WAN but using the Gateway IP, and not an IPPool and with the addition of a session-helper, we were able to establish a connection. 

But I am still a bit perplexed as to why this doesn't seem to affect my SFTP connection... I am unsure if it is related to the target SFTP server or simply the design of SFTP, but I can use my original configuration(SD-wan to NAT IPPOOL) outbound to the server and I can establish a connection without any issue. Still mind boggling...

So for now, specific rule for FTP to use Gateway IP, SFTP uses my normal firewall rule, and FTPs is still a pain in my behind... I sent a pcap to the engineer from a working connection(testing ISP line, bypassing Fortigate) and will see what he can discover on his end. The traces we ran through the Fortigate clearly showed SNAT and DNAT , original and reply which should indicate connectivity between the two. It simply doesn't allow the directory listing and then times out. Here is part of a connection :

[size="1"]id=20085 trace_id=3065 func=print_pkt_detail line=5363 msg="vd-root received a packet(proto=6, 10.1.99.243:58607->FTPs Server:990) from LAN-LACP-P39P40. flag [.], seq 2921097277, ack 2622610569, win 64240"[/size] id=20085 trace_id=3065 func=resolve_ip_tuple_fast line=5438 msg="Find an existing session, id-01742fd2, original direction" id=20085 trace_id=3065 func=npu_handle_session44 line=1079 msg="Trying to offloading session from LAN-LACP-P39P40 to port17, skb.npu_flag=00000400 ses.state=04000200 ses.npu_state=0x00000001" id=20085 trace_id=3065 func=__ip_session_run_tuple line=3164 msg="SNAT 10.1.99.243->External ISP GW:58607" id=20085 trace_id=3065 func=__ip_session_run_tuple line=3215 msg="run helper-ftp(dir=original)" [size="1"]id=20085 trace_id=3066 func=print_pkt_detail line=5363 msg="vd-root received a packet(proto=6, FTPs Server:990->External ISP GW:58585) from port17. flag [.], seq 2487703257, ack 3320002037, win 32768"[/size] id=20085 trace_id=3066 func=resolve_ip_tuple_fast line=5438 msg="Find an existing session, id-0173ff47, reply direction" id=20085 trace_id=3066 func=__ip_session_run_tuple line=3178 msg="DNAT External ISP GW:58585->10.1.99.243:58585" id=20085 trace_id=3066 func=npu_handle_session44 line=1079 msg="Trying to offloading session from port17 to LAN-LACP-P39P40, skb.npu_flag=00000000 ses.state=04000200 ses.npu_state=0x00000001" id=20085 trace_id=3066 func=__ip_session_run_tuple line=3215 msg="run helper-ftp(dir=reply)" [size="1"]id=20085 trace_id=3067 func=print_pkt_detail line=5363 msg="vd-root received a packet(proto=6, FTPs Server:990->External ISP GW:58607) from port17. flag [.], seq 2622610569, ack 2921097277, win 4096"[/size] id=20085 trace_id=3067 func=resolve_ip_tuple_fast line=5438 msg="Find an existing session, id-01742fd2, reply direction" id=20085 trace_id=3067 func=__ip_session_run_tuple line=3178 msg="DNAT External ISP GW:58607->10.1.99.243:58607" id=20085 trace_id=3067 func=npu_handle_session44 line=1079 msg="Trying to offloading session from port17 to LAN-LACP-P39P40, skb.npu_flag=00000400 ses.state=04000200 ses.npu_state=0x00000001" id=20085 trace_id=3067 func=__ip_session_run_tuple line=3215 msg="run helper-ftp(dir=reply)" [size="1"]id=20085 trace_id=3068 func=print_pkt_detail line=5363 msg="vd-root received a packet(proto=6, 10.1.99.243:58607->FTPs Server:990) from LAN-LACP-P39P40. flag [.], seq 2921097277, ack 2622610569, win 64240"[/size] id=20085 trace_id=3068 func=resolve_ip_tuple_fast line=5438 msg="Find an existing session, id-01742fd2, original direction" id=20085 trace_id=3068 func=npu_handle_session44 line=1079 msg="Trying to offloading session from LAN-LACP-P39P40 to port17, skb.npu_flag=00000400 ses.state=04000200 ses.npu_state=0x00000001" id=20085 trace_id=3068 func=__ip_session_run_tuple line=3164 msg="SNAT 10.1.99.243->External ISP GW:58607" id=20085 trace_id=3068 func=__ip_session_run_tuple line=3215 msg="run helper-ftp(dir=original)" [size="1"]id=20085 trace_id=3069 func=print_pkt_detail line=5363 msg="vd-root received a packet(proto=6, FTPs Server:990->External ISP GW:58607) from port17. flag [.], seq 2622610569, ack 2921097503, win 65309"[/size] id=20085 trace_id=3069 func=resolve_ip_tuple_fast line=5438 msg="Find an existing session, id-01742fd2, reply direction" id=20085 trace_id=3069 func=__ip_session_run_tuple line=3178 msg="DNAT External ISP GW:58607->10.1.99.243:58607" id=20085 trace_id=3069 func=npu_handle_session44 line=1079 msg="Trying to offloading session from port17 to LAN-LACP-P39P40, skb.npu_flag=00000400 ses.state=04000200 ses.npu_state=0x00000001" id=20085 trace_id=3069 func=__ip_session_run_tuple line=3215 msg="run helper-ftp(dir=reply)"

[bcote]

So quick update, the FTPs connection would simply not complete with our external party. Most likely using an archaic server of some sort that simply will not work with a session-helper and needs port-forwarding. Although myself and the engineer did not want to go that route, since I can limit where the traffic is coming from, I proceeded to do a VIP with port forwarding and the connection was completed properly. 

For whatever reason, government agencies enjoy leaving systems with age old designs and do not feel like doing any type of update, even if it would secure their traffic even more. 

Anyways, everything on my end is now working, it simply took going through a couple of hoops to achieve it.

Thanks for the help,

Ben