Troubleshooting Tip: How to troubleshoot for event handler related issues

Description

This article describes various ways to investigate and troubleshoot for FortiAnalyzer event handler-related issues.

Scope

FortiAnalyzer v7.4.0, v7.2.2 & above.

Solution

Note: In this article, 'Action=login' FortiGate event logs will be used to trigger the FortiAnalyzer event.

1. Verify that the logs are received and visible under FortiAnalyzer -> Log View.
   

Once FortiAnalyzer has received the logs, the handler will trigger the event based on the handler rule and threshold settings:

2. Verify fazalertd and sqllogd daemons are still running by running the following command:

faz# diagnose test app sqllogd 1

faz# diagnose test app fazalertd 1 -> fazalertd introduced starting from v7.2.2 and above.

3. Verify whether the logs have hit the handler rule:

faz# diagnose test app sqllogd 200 conf adom=<adom-name> handler=<handler-name>

Example output:

faz# diagnose test app sqllogd 200 conf adom=test-adom handler="test-handler"

* Enabled rules in Adom test-adom [205] is 400:
----------------------------------------
Handler Name : test-handler/1780784373
Rule Name : test-rule/2608794398
Handler Type : Basic
Data Src Type : memory
Selector Name : test-selector
Log chk/hit : 4/2 <----- Verify whether the hit count increases after receiving the log.
Instance/AP/Sum: 1/1/1
Filterkey : 3869101813645295628

4. Perform live debugging and verify whether the logs are triggering any event:

To enable live debugging:

faz# diagnose test app fazalertd 200 debug -> Ensure it shows 'debug is on', else run the command again.

faz# diagnose debug enable

Replicate the event (e.g, login from FortiGate). Verify and analyze the debug output.

To disable live debugging:

faz# diagnose test app fazalertd 200 debug

faz# diagnose debug disable

Sample output (when the event is triggered based on the log):

5. Try restarting the sqllogd & fazalertd daemon, and verify the results again:

faz# diagnose test app sqllogd 99

faz# diagnose test app fazalertd 99

6. If the commands above do not help, try the following commands to force restart the fazalertd daemon and verify the results again.
   Note: This will reset all correlated event handler data, such as threshold window time.

faz# diagnose test app sqllogd 200 debug rocksdb reset fazalertd

Warning: This will reset fazalertd rocksdb and restart fazalertd,
execute the command again in one minute to reset fazalertd rocksdb.

faz#

faz# diagnose test app sqllogd 200 debug rocksdb reset fazalertd

The fazalertd rocksdb was reset

7. Troubleshooting Event Generation Failure.

Send the corresponding information in the ticket:

• Config of FortiAnalyzer.
• Raw log of FortiAnalyzer.
• execute tac report.

diagnose test connection mailserver <mailserver> <source SMTP address> <destination SMTP address>

The following commands on the FortiAnalyzer will provide more information regarding the SMTP client application.

For FortiManager/FortiAnalyzer v7.6 or above, perform a flow capture:

diagnose test application fazmaild ?

<Integer> Debug level (08).

diagnose debug application fazmaild 8

diagnose debug timestamp enable

diagnose debug enable

diagnose debug disable <- To stop it.

diagnose debug reset

In the FortiAnalyzer, enter the following commands while running a 'diagnose log test' action in the FortiGate CLI:

diagnose test application sqllogd 200

diagnose test application sqllogd 200 status

diagnose test application sqllogd 200 config

diagnose debug application sqllogd 8

diagnose debug enable

diagnose debug application fazmaild 255

diagnose debug disable

diagnose debug reset

Related articles:

Technical Tip: How to Validate Event Handler in FortiManager and FortiAnalyzer

Technical Tip: How to set up Email Notifications with notification.fortinet.net

Troubleshooting Tip: How to understand the email SMTP issues and its causes