Troubleshooting Tip: How to troubleshoot for event handler related issues

[tnesh]

Description

This article describes various ways to investigate and troubleshoot for FortiAnalyzer event handler-related issues.

Scope

FortiAnalyzer v7.4.0, v7.2.2 & above.

Solution

Note: In this article, 'Action=login' FortiGate event logs will be used to trigger the FortiAnalyzer event.

1. Verify that the logs are received and visible under FortiAnalyzer -> Log View.
   

   

   Once FortiAnalyzer received the logs, the handler will trigger the event based on the handler rule and threshold settings:
   
   

   

2. Verify fazalertd and sqllogd daemon are still running by running the below command:
   

   
   faz# diag test app sqllogd 1

   faz# diag test app fazalertd 1 -> fazalertd introduced starting v7.2.2 & above.

3. Verify whether the logs have hit the handler rule:
   
   

   faz# diag test app sqllogd 200 conf adom=<adom-name> handler=<handler-name>

   Sample output:

   faz# diag test app sqllogd 200 conf adom=test-adom handler="test-handler"

   * Enabled rules in Adom test-adom [205] is 400:
   ----------------------------------------
   Handler Name : test-handler/1780784373
   Rule Name : test-rule/2608794398
   Handler Type : Basic
   Data Src Type : memory
   Selector Name : test-selector
   Log chk/hit : 4/2 -> verify whether the hit count increases after receiving the log.
   Instance/AP/Sum: 1/1/1
   Filterkey : 3869101813645295628
   
   

4. Perform live debugging and verify whether the logs are triggering any event:
   
   

   To enable live debugging:

   faz# diag test app fazalertd 200 debug -> ensure it shows 'debug is on', else run the command again.

   faz# diag debug enable

   . . . Replicate the event (Eg: Login from FortiGate). Verify and analyze the debug output.

   To disable live debug:

   faz# diag test app fazalertd 200 debug

   faz# diag debug disable

   Sample output (when the event is triggered based on the log):
   
   

   

5. Try restarting sqllogd & fazalertd daemon, and verify the results again:
   
   

   faz# diag test app sqllogd 99

   faz# diag test app fazalertd 99

6. If the above command does not help, try the following commands to force restart fazalertd daemon and verify the results again.
   Note: This will reset all correlated event handler data such as threshold window time.
   
   

   faz# diag test app sqllogd 200 debug rocksdb reset fazalertd

   Warning: This will reset fazalertd rocksdb and restart fazalertd,
   execute the command again in one minute to reset fazalertd rocksdb.
   

   faz#

   faz# diag test app sqllogd 200 debug rocksdb reset fazalertd

   The fazalertd rocksdb was reset
   

Related article:

Technical Tip: How to Validate Event Handler in FortiManager and FortiAnalyzer