Troubleshooting Tip: Debug flow flags

Description

This article describes the TCP flags when using the debug flow (Technical Tip: Using filters to review traffic traversing the FortiGate).

Scope

FortiGate.

Solution

Example:

SYN from user:

12:45:15 id=65308 trace_id=117 func=print_pkt_detail line=5879 msg="vd-root:0 received a packet(proto=6, 192.168.1.204:63853->213.13.146.142:443) tun_id=0.0.0.0 from internal. flag [S], seq 2963120482, ack 0, win 64240"
12:45:15 id=65308 trace_id=117 func=init_ip_session_common line=6063 msg="allocate a new session-00f0defa"
12:45:15 id=65308 trace_id=117 func=iprope_dnat_check line=5474 msg="in-[internal], out-[]"
...
12:45:15 id=65308 trace_id=117 func=vf_ip_route_input_common line=2612 msg="find a route: flag=04000000 gw-85.245.xyz.1 via wan1"
12:45:15 id=65308 trace_id=117 func=__iprope_fwd_check line=807 msg="in-[internal], out-[wan1], skb_flags-02000000, vid-0, app_id: 0, url_cat_id: 0"
...
12:45:15 id=65308 trace_id=117 func=get_new_addr line=1265 msg="find SNAT: IP-85.xyz.xyz.xyz(from IPPOOL), port-63853"
12:45:15 id=65308 trace_id=117 func=__iprope_check_one_policy line=2365 msg="policy-2 is matched, act-accept"

Allowed by policy ID 2:

12:45:15 id=65308 trace_id=117 func=fw_forward_handler line=987 msg="Allowed by Policy-2: SNAT"
12:45:15 id=65308 trace_id=117 func=__ip_session_run_tuple line=3429 msg="SNAT 192.168.1.204->85.xyz.xyz.xyz:63853"
...

Server SYN/ACK:

id=65308 trace_id=118 func=print_pkt_detail line=5879 msg="vd-root:0 received a packet(proto=6, 213.13.146.142:443->85.xyz.xyz.xyz:63853) tun_id=0.0.0.0 from wan1. flag [S.], seq 3369286322, ack 2963120483, win 14600"
id=65308 trace_id=118 func=resolve_ip_tuple_fast line=5967 msg="Find an existing session, id-00f0defa, reply direction"
id=65308 trace_id=118 func=__ip_session_run_tuple line=3442 msg="DNAT 85.xyz.xyz.xyz:63853->192.168.1.204:63853"
id=65308 trace_id=118 func=vf_ip_route_input_common line=2612 msg="find a route: flag=00000000 gw-192.168.1.204 via internal"
id=65308 trace_id=118 func=npu_handle_session44 line=1224 msg="Trying to offloading session from wan1 to internal, skb.npu_flag=00000400 ses.state=00012204 ses.npu_state=0x00003094"
id=65308 trace_id=118 func=np6xlite_fos_set_nturbo_ips_fwd_session line=626 msg="push nturbo session oid 12"
id=65308 trace_id=118 func=ip_session_install_npu_session line=364 msg="npu session installation succeeded"
id=65308 trace_id=118 func=fw_forward_dirty_handler line=442 msg="state=00012204, state2=00000001, npu_state=00003894"
id=65308 trace_id=118 func=np6xlite_hif_nturbo_build_vtag line=1233 msg="vtag->magic d153beef, vtag->coretag 77, vtag->vid 0

User ACK:

id=65308 trace_id=119 func=print_pkt_detail line=5879 msg="vd-root:0 received a packet(proto=6, 192.168.1.204:63853->213.13.146.142:443) tun_id=0.0.0.0 from internal. flag [.], seq 2963120483, ack 3369286323, win 1026"
id=65308 trace_id=119 func=resolve_ip_tuple_fast line=5967 msg="Find an existing session, id-00f0defa, original direction"
id=65308 trace_id=119 func=npu_handle_session44 line=1224 msg="Trying to offloading session from internal to wan1, skb.npu_flag=00000400 ses.state=00012204 ses.npu_state=0x00003894"
id=65308 trace_id=119 func=np6xlite_fos_set_nturbo_ips_fwd_session line=626 msg="push nturbo session oid 12"
id=65308 trace_id=119 func=ip_session_install_npu_session line=364 msg="npu session installation succeeded"

A FIN-ACK ACK from the user to end the session:

trace_id=158 func=print_pkt_detail line=5879 msg="vd-root:0 received a packet(proto=6, 192.168.1.204:63853->213.13.146.142:443) tun_id=0.0.0.0 from internal. flag [F.], seq 2963123076, ack 3369291249, win 1026"
trace_id=158 func=resolve_ip_tuple_fast line=5967 msg="Find an existing session, id-00f0defa, original direction"

...

A demo file is attached to this article.

Reset short example:

id=65308 trace_id=927 func=print_pkt_detail line=5802 msg="vd-root:0 received a packet(proto=6, 10.10.10.10:56664->20.20.20.20:443) tun_id=0.0.0.0 from wan2. flag [R.], seq 3037276435, ack 422986220, win 501"
id=65308 trace_id=927 func=resolve_ip_tuple_fast line=5890 msg="Find an existing session, id-0f96ffb9, original direction"

id=65308 trace_id=928 func=print_pkt_detail line=5802 msg="vd-root:0 received a packet(proto=6, 192.168.38.17:80->10.10.10.10:56664) tun_id=0.0.0.0 from local. flag [.], seq 422986220, ack 3037276435, win 85"
id=65308 trace_id=928 func=resolve_ip_tuple_fast line=5890 msg="Find an existing session, id-0f96ffb9, reply direction"

id=65308 trace_id=942 func=print_pkt_detail line=5802 msg="vd-root:0 received a packet(proto=6, 10.10.10.10:56664->20.20.20.20:443) tun_id=0.0.0.0 from wan2. flag [R], seq 3037276435, ack 0, win 0"
id=65308 trace_id=942 func=resolve_ip_tuple_fast line=5890 msg="Find an existing session, id-0f96ffb9, original direction"

Typical reasons for seeing a TCP RST:

• A packet arrives for a session the host no longer has (because it timed out, was closed, the app restarted, NAT state changed, etc.).
• A packet arrives for a session the host no longer has (because it timed out, was closed, the app restarted, NAT state changed, etc.).
• Protocol/port mismatch (very common). The client thinks it is connecting to HTTPS/TLS on 443, but the other side actually responds with plain HTTP (or something else). Many clients will abort with an RST upon receiving unexpected data.

Related documents:

CLI Troubleshooting Cheat Sheet

Technical Tip: FortiGate Resource Lists

Troubleshooting Tip: First steps to troubleshoot connectivity problems to or through a FortiGate with sniffer, debug flow, session list, routing table

tcpdump101.com: This tool can help craft debug flow or diag sniffer packet CLI.