Description
This article describes various ways to investigate and troubleshoot for FortiAnalyzer event handler-related issues.
Scope
FortiAnalyzer v7.4.0, v7.2.2 & above.
Solution
Note: In this article, 'Action=login' FortiGate event logs will be used to trigger the FortiAnalyzer event.
- Verify that the logs are received and visible under FortiAnalyzer -> Log View.
Once FortiAnalyzer has received the logs, the handler will trigger the event based on the handler rule and threshold settings:
- Verify fazalertd and sqllogd daemon are still running by running the following command:
faz# diag test app sqllogd 1
faz# diag test app fazalertd 1 -> fazalertd introduced starting from v7.2.2 & above.
- Verify whether the logs have hit the handler rule:
faz# diagnose test app sqllogd 200 conf adom=<adom-name> handler=<handler-name>
Example output:
faz# diagnose test app sqllogd 200 conf adom=test-adom handler="test-handler"
* Enabled rules in Adom test-adom [205] is 400:
----------------------------------------
Handler Name : test-handler/1780784373
Rule Name : test-rule/2608794398
Handler Type : Basic
Data Src Type : memory
Selector Name : test-selector
Log chk/hit : 4/2 <----- Verify whether the hit count increases after receiving the log.
Instance/AP/Sum: 1/1/1
Filterkey : 3869101813645295628
- Perform live debugging and verify whether the logs are triggering any event:
To enable live debugging:
faz# diagnose test app fazalertd 200 debug -> ensure it shows 'debug is on', else run the command again.
faz# diagnose debug enable
Replicate the event (e.g: Login from FortiGate). Verify and analyze the debug output.
To disable live debugging:
faz# diag test app fazalertd 200 debug
faz# diag debug disable
Sample output (when the event is triggered based on the log):
- Try restarting the sqllogd & fazalertd daemon, and verify the results again:
faz# diagnose test app sqllogd 99
faz# diagnose test app fazalertd 99
- If the commands above do not help, try the following commands to force restart the fazalertd daemon and verify the results again.
Note: This will reset all correlated event handler data such as threshold window time.
faz# diagnose test app sqllogd 200 debug rocksdb reset fazalertd
Warning: This will reset fazalertd rocksdb and restart fazalertd,
execute the command again in one minute to reset fazalertd rocksdb.
faz#
faz# diagnose test app sqllogd 200 debug rocksdb reset fazalertd
The fazalertd rocksdb was reset
- Troubleshooting Event Generation Failure.
Send the corresponding information in the ticket:
- Config of FortiAnalyzer.
- Raw log of FortiAnalyzer.
- exe tac report
diagnose test connection mailserver <mailserver> <source SMTP address> <destination SMTP address>
The following commands on the FortiAnalyzer will provide more information regarding the SMTP client application.
For FortiManager / FortiAnalyzer 7.6 or above, perform a flow capture:
diagnose test application fazmaild ?
<Integer> Debug level (08).
diagnose debug application fazmaild 8
diagnose debug timestamp enable
diagnose debug enable
diagnose debug disable <- To stop it.
diagnose debug reset
In the FortiAnalyzer, enter the following commands while running a 'diag log test' action in the FortiGate CLI:
diagnose test application sqllogd 200
diagnose test application sqllogd 200 status
diagnose test application sqllogd 200 config
diagnose debug application sqllogd 8
diagnose debug enable
diagnose debug application fazmaild 255
diagnose debug disable
diagnose debug reset
Related article:
Technical Tip: How to Validate Event Handler in FortiManager and FortiAnalyzer
