Description
This article provides an explanation of the entry 'action=ip-conn' that may be seen in the traffic logs.
For example:
Aug 23 03:52:14 10.95.216.1 date=2016-08-23 time=03:52:14 devname=external-fgt-01 devid=FGXXXXXXXX logid=0000000011 type=traffic subtype=forward level=warning vd=root srcip=1.2.3.4 srcport=48641 srcintf="PUBLIC-VIP" dstip=4.3.2.1 dstport=80 dstintf="LOCAL-PORT" poluuid=342f44-adff-asdfasd-mujjh-5yghnhn56hhd sessionid=3025325172 proto=6 action=ip-conn policyid=2 appcat="unscanned" crscore=5 craction=262144 crlevel=low
Scope
FortiGate.
Solution
The value 'ip-conn' in the log field description means that traffic was allowed, but the session closed as the FortiGate did not receive any reply packet. The resulting error shows as 'IP connection error'.
This can occur if the connection to the remote server fails or a timeout occurs. This usually occurs on the internet segment (FortiGate to ISP/server), and most times it is not caused by FortiGate.
Packet losses may be experienced due to a bad connection, traffic congestion, or high memory and CPU utilization (on either FortiGate or the remote host).
To troubleshoot this issue, run an extended ping test from the host to the remote server to see if packet losses will be experienced.
The same can be repeated from the FortiGate to the remote server (to exclude the LAN segment as a possible fault point):
execute ping-options repeat-count 10000
execute ping <IP.OF.REMOTE.SERVER>
Check the resource utilization on the FortiGate and perform the equivalent on the host:
diagnose hardware sysinfo memory
get system performance status
get system performance top
diagnose system top
Run the following packet sniffer in the FortiGate CLI to quickly check if all packets are replied to:
diagnose sniffer packet any "host x.x.x.x" 4 0 a <----- Where x.x.x.x is the IP of the remote server
additionally, a filter for specific ports can be added here:
diagnose sniffer packet any "host x.x.x.x and port 53" 4 0 a
For TCP connections it is better to run a full capture;
diagnose sniffer packet any "host x.x.x.x" 6 0 a <----- Where x.x.x.x is the IP of the remote server
This error can also be caused if an ICMP error message is sent by a client or server that hits this session. In the following example, a client is using the same source port (12345) to make two consecutive queries to two different DNS servers:
2024-06-26 16:15:27.516 0.000 192.168.5.4 12345 DNS 8.8.8.8 67 0x0001 (1) Standard query 0x0000 A sapo.pt
2024-06-26 16:15:27.524 0.007 192.168.5.4 12345 DNS 8.8.4.4 67 0x0001 (1) Standard query 0x0000 A sapo.pt
2024-06-26 16:15:27.527 0.003 8.8.8.8 53 DNS 192.168.5.4 83 0x8a1f (35359) Standard query response 0x0000 A sapo.pt A 213.13.146.142
2024-06-26 16:15:27.537 0.009 8.8.4.4 53 DNS 192.168.5.4 83 0x70bb (28859) Standard query response 0x0000 A sapo.pt A 213.13.146.142
2024-06-26 16:15:27.538 0.000 192.168.5.4 53 ICMP 8.8.4.4 111 0x8c8d (35981),0x70bb (28859) Destination unreachable (Port unreachable)
When the response for the second query arrives back to the host, the host is no longer listening on port 12345, so it generates a ICMP error message 'Type 3 Destination Unreachable, Code 3 Port Unreachable' and sends it back to the server. This will be logged as action="ip-conn":
date=2024-06-26 time=16:18:28 eventtime=1719415108003787879 tz="+0100" logid="0000000011" type="traffic" subtype="forward" level="warning" vd="PROD" srcip=192.168.5.4 srcport=12345 srcintf="internal" srcintfrole="undefined" dstip=8.8.4.4 dstport=53 dstintf="npu0_vlink1" dstintfrole="undefined" srccountry="Reserved" dstcountry="United States" sessionid=6250861 proto=17 action="ip-conn" policyid=20 policytype="policy" poluuid="464109a4-25a5-51ef-4b07-e23a9121d904" policyname="dns" service="DNS" appcat="unscanned" crscore=5 craction=262144 crlevel="low" devtype="Computer" osname="Debian" mastersrcmac="b6:e0:31:37:ca:73" srcmac="b6:e0:31:37:ca:73" srcserver=0
