Technical Tip: How to troubleshoot FortiGate and FortiSandbox communication

[fgilloteau_FTNT]

Description
This article gives some useful information and troubleshooting commands related to FortiGate/FortiSandbox communication.

Solution
On the FortiGate:

Connectivity:

# execute system fortisandbox test-connectivity

Return status should be “Reachable” otherwise a TCP connection on dstport 514 can not be established.

Process debug:

# diag debug application quarantine -1
# diag debug enable

Some errors messages which can be revealed:

Example 1:

2019-01-07 10:10:42 quar_remote_connect()-745: oftp_connect failed: connect() failed: Connection refused.

-> TCP connection to port 514 on the target IP cannot established

Note: If case of a cluster of FortiSandbox Active-Passive, make sure the FortiGate is configured to reach only the virtual IP address of the FortiSandbox cluster, as only the MASTER FortiSandbox can receive the traffic

Example 2:

2019-01-17 09:54:47 __check_dev_tasks()-788: req-4392648 is deleted: ttl=122389, xfer_retry=0
2019-01-17 09:54:47 quar_put_job_req()-330: Job 4392648 deleted
2019-01-17 09:54:47 __check_dev_tasks()-788: req-4392654 is deleted: ttl=121628, xfer_retry=0
2019-01-17 09:54:47 quar_put_job_req()-330: Job 4392654 deletec

-> A job can stay in the quard queue maximum 20 minutes, then it is deleted (120000 ms)
-> Considering the log above, it means that a job could not be sent after 20 minutes, possibly because the FortiSandbox is unreachable

Process stats:

# diag test application quarantined 2

Quarantine daemon state:
QUAR mem: mem_used=1273, mem_limit=255915, threshold=191934
dropped(24825 by quard, 3032 by callers)
pending-jobs=61, tot-mem=655, last_ipc_run=17, check_new_req=1
alloc_job_failed=0, job_wrong_type=0, job_wrong_req_len=0, job_invalid_qfd=0
tgz_create_failed=0, tgz_attach_failed=0, qfd_mmap_failed=0, buf_attached=21
xfer-fas:
    ips: total=0, handled=0, accepted=0
    quar: total=0, handled=0, accepted=0
    archive: total=0, handled=0, accepted=0
    analytics: total=0, handled=0, accepted=0, local_dups=0
    analytics stats: total=0, handled=0, accepted=0
    last_rx=0, last_tx=0, error_rx=0, error_tx=0
    num_tasks=0, mem_used=0, xfer_status=0
fortisandbox-fsb1:
    ips: total=0, handled=0, accepted=0
    quar: total=0, handled=0, accepted=0
    archive: total=0, handled=0, accepted=0
    analytics: total=212704, handled=210956, accepted=428, local_dups=1748
    analytics stats: total=3850, handled=3850, accepted=4
    last_rx=270494671, last_tx=270494671, error_rx=20, error_tx=0
    num_tasks=12, mem_used=47, xfer_status=0
    buf_len=0, buf_pos=0
fortisandbox-fsb2:
    ips: total=0, handled=0, accepted=0
    quar: total=0, handled=0, accepted=0
    archive: total=0, handled=0, accepted=0
    analytics: total=212436, handled=210599, accepted=422, local_dups=1837
    analytics stats: total=3718, handled=3718, accepted=1
    last_rx=270494671, last_tx=270494671, error_rx=5, error_tx=0
    num_tasks=9, mem_used=105, xfer_status=0
    buf_len=0, buf_pos=0
fortisandbox-fsb3:
    ips: total=0, handled=0, accepted=0
    quar: total=0, handled=0, accepted=0
    archive: total=0, handled=0, accepted=0
    analytics: total=212328, handled=210456, accepted=422, local_dups=1872
    analytics stats: total=3784, handled=3784, accepted=3
    last_rx=270494975, last_tx=270494975, error_rx=6, error_tx=0
    num_tasks=9, mem_used=8, xfer_status=0
    buf_len=0, buf_pos=0
fortisandbox-fsb4:
    ips: total=0, handled=0, accepted=0
    quar: total=0, handled=0, accepted=0
    archive: total=0, handled=0, accepted=0
    analytics: total=212019, handled=210221, accepted=436, local_dups=1798
    analytics stats: total=3792, handled=3792, accepted=3
    last_rx=270494888, last_tx=270494888, error_rx=15, error_tx=0
    num_tasks=8, mem_used=164, xfer_status=0
    buf_len=0, buf_pos=0
fortisandbox-fsb5:
    ips: total=0, handled=0, accepted=0
    quar: total=0, handled=0, accepted=0
    archive: total=0, handled=0, accepted=0
    analytics: total=211689, handled=209945, accepted=409, local_dups=1744
    analytics stats: total=3679, handled=3679, accepted=0
    last_rx=270495071, last_tx=270495071, error_rx=1, error_tx=0
    num_tasks=12, mem_used=197, xfer_status=0
    buf_len=0, buf_pos=0
fortisandbox-fsb6:
    ips: total=0, handled=0, accepted=0
    quar: total=0, handled=0, accepted=0
    archive: total=0, handled=0, accepted=0
    analytics: total=211954, handled=209958, accepted=364, local_dups=1996
    analytics stats: total=3716, handled=3716, accepted=0
    last_rx=270494975, last_tx=270494975, error_rx=25, error_tx=0
    num_tasks=11, mem_used=480, xfer_status=0
    buf_len=0, buf_pos=0
global-faz:
    ips: total=0, handled=0, accepted=0
    quar: total=0, handled=0, accepted=0
    archive: total=0, handled=0, accepted=0
    analytics: total=0, handled=0, accepted=0, local_dups=0
    analytics stats: total=0, handled=0, accepted=0
    last_rx=0, last_tx=0, error_rx=0, error_tx=0
    num_tasks=0, mem_used=0, xfer_status=0

Configurations:

On the FortiGate:

# config antivirus profile
    edit <myprofile>
        set ftgd-analytics everything

-> This sends everything to the FortiSandbox, so can impact the performance potentially

        set analytics-max-upload 10

-> The file size configured here can also impact the performance. Using default value is recommended

        set analytics-wl-filetype 1

-> This can limit some file type extensions to send only some files extension (.js, .exe) to the FortiSandbox


On the FortiSandbox:

1) Config

To check IP configuration:

# show

To check HA:

# hc-settings -l
# hc-status –l

2) Authorized devices

Check also that the device is “Authorized” in the GUI menu Scan Input -> Device

3) Traffic

# tcpdump -c 1000 port 514      <--------- This will capture 1000 packets

4) Process and CPU

# diagnose-sys-top

5) CPU, Memory and scanning statistics

# diagnose-syst-perf

6) Queue

# pending-jobs show all all