Technical Tip: How to troubleshoot FortiGate and FortiSandbox communication

Description

This article describes useful information and troubleshooting commands related to FortiGate/FortiSandbox communication.

Scope

FortiGate.

Solution

On the FortiGate:

Connectivity:

execute system fortisandbox test-connectivity

The return status should be 'Reachable' otherwise a TCP connection on dstport 514 can not be established.

Process debug:

diagnose debug disable

diagnose debug reset

diagnose debug application quarantine -1
diagnose debug enable

Some errors messages which can be revealed:

Example 1:

2019-01-07 10:10:42 quar_remote_connect()-745: oftp_connect failed: connect() failed: Connection refused. <----- TCP connection to port 514 on the target IP cannot established.

Note: If case of a cluster of FortiSandbox Active-Passive, make sure the FortiGate is configured to reach only the virtual IP address of the FortiSandbox cluster, as only the MASTER FortiSandbox can receive the traffic.

Example 2:

2019-01-17 09:54:47 __check_dev_tasks()-788: req-4392648 is deleted: ttl=122389, xfer_retry=0
2019-01-17 09:54:47 quar_put_job_req()-330: Job 4392648 deleted
2019-01-17 09:54:47 __check_dev_tasks()-788: req-4392654 is deleted: ttl=121628, xfer_retry=0
2019-01-17 09:54:47 quar_put_job_req()-330: Job 4392654 deleted

• A job can stay in the quard queue for a maximum of 20 minutes, then it is deleted (120000 ms).
• Considering the log above, it means that a job could not be sent after 20 minutes, possibly because the FortiSandbox is unreachable
  
  

Process stats:

diagnose test application quarantined 2

Quarantine daemon state:
QUAR mem: mem_used=1273, mem_limit=255915, threshold=191934
dropped(24825 by quard, 3032 by callers)
pending-jobs=61, tot-mem=655, last_ipc_run=17, check_new_req=1
alloc_job_failed=0, job_wrong_type=0, job_wrong_req_len=0, job_invalid_qfd=0
tgz_create_failed=0, tgz_attach_failed=0, qfd_mmap_failed=0, buf_attached=21
xfer-fas:
    ips: total=0, handled=0, accepted=0
    quar: total=0, handled=0, accepted=0
    archive: total=0, handled=0, accepted=0
    analytics: total=0, handled=0, accepted=0, local_dups=0
    analytics stats: total=0, handled=0, accepted=0
    last_rx=0, last_tx=0, error_rx=0, error_tx=0
    num_tasks=0, mem_used=0, xfer_status=0
fortisandbox-fsb1:
    ips: total=0, handled=0, accepted=0
    quar: total=0, handled=0, accepted=0
    archive: total=0, handled=0, accepted=0
    analytics: total=212704, handled=210956, accepted=428, local_dups=1748
    analytics stats: total=3850, handled=3850, accepted=4
    last_rx=270494671, last_tx=270494671, error_rx=20, error_tx=0
    num_tasks=12, mem_used=47, xfer_status=0
    buf_len=0, buf_pos=0
fortisandbox-fsb2:
    ips: total=0, handled=0, accepted=0
    quar: total=0, handled=0, accepted=0
    archive: total=0, handled=0, accepted=0
    analytics: total=212436, handled=210599, accepted=422, local_dups=1837
    analytics stats: total=3718, handled=3718, accepted=1
    last_rx=270494671, last_tx=270494671, error_rx=5, error_tx=0
    num_tasks=9, mem_used=105, xfer_status=0
    buf_len=0, buf_pos=0
fortisandbox-fsb3:
    ips: total=0, handled=0, accepted=0
    quar: total=0, handled=0, accepted=0
    archive: total=0, handled=0, accepted=0
    analytics: total=212328, handled=210456, accepted=422, local_dups=1872
    analytics stats: total=3784, handled=3784, accepted=3
    last_rx=270494975, last_tx=270494975, error_rx=6, error_tx=0
    num_tasks=9, mem_used=8, xfer_status=0
    buf_len=0, buf_pos=0
fortisandbox-fsb4:
    ips: total=0, handled=0, accepted=0
    quar: total=0, handled=0, accepted=0
    archive: total=0, handled=0, accepted=0
    analytics: total=212019, handled=210221, accepted=436, local_dups=1798
    analytics stats: total=3792, handled=3792, accepted=3
    last_rx=270494888, last_tx=270494888, error_rx=15, error_tx=0
    num_tasks=8, mem_used=164, xfer_status=0
    buf_len=0, buf_pos=0
fortisandbox-fsb5:
    ips: total=0, handled=0, accepted=0
    quar: total=0, handled=0, accepted=0
    archive: total=0, handled=0, accepted=0
    analytics: total=211689, handled=209945, accepted=409, local_dups=1744
    analytics stats: total=3679, handled=3679, accepted=0
    last_rx=270495071, last_tx=270495071, error_rx=1, error_tx=0
    num_tasks=12, mem_used=197, xfer_status=0
    buf_len=0, buf_pos=0
fortisandbox-fsb6:
    ips: total=0, handled=0, accepted=0
    quar: total=0, handled=0, accepted=0
    archive: total=0, handled=0, accepted=0
    analytics: total=211954, handled=209958, accepted=364, local_dups=1996
    analytics stats: total=3716, handled=3716, accepted=0
    last_rx=270494975, last_tx=270494975, error_rx=25, error_tx=0
    num_tasks=11, mem_used=480, xfer_status=0
    buf_len=0, buf_pos=0
global-faz:
    ips: total=0, handled=0, accepted=0
    quar: total=0, handled=0, accepted=0
    archive: total=0, handled=0, accepted=0
    analytics: total=0, handled=0, accepted=0, local_dups=0
    analytics stats: total=0, handled=0, accepted=0
    last_rx=0, last_tx=0, error_rx=0, error_tx=0
    num_tasks=0, mem_used=0, xfer_status=0

Configurations:

On the FortiGate:

config antivirus profile
    edit <myprofile>
        set ftgd-analytics everything

• This sends everything to the FortiSandbox, so it may impact the performance.

set analytics-max-upload 10

• The file size configured here can also impact the performance. Using the default value is recommended,

set analytics-wl-filetype 1

• This can limit some file type extensions to send only some files extension (.js, .exe) to the FortiSandbox
  
  

On the FortiSandbox:

1. Config: 
   
   

To check IP configuration:

show

To check HA:

hc-settings -l
hc-status –l

2. Authorized devices:

Additionally, check that the device is 'Authorized' in the GUI under Scan Input -> Device.

3. Traffic.

tcpdump -c 1000 port 514      <----- This will capture 1000 packets.

4. Process and CPU.

diagnose-sys-top

5. CPU, Memory and scanning statistics.

diagnose-syst-perf

6. Queue.

pending-jobs show all all