Solution |
Scenario:
- The customer was getting an error at 2% (loading too slow) in GUI while checking for the forward traffic logs in GUI.
- Changed the status to memory but still no logs same error.
- Disabled the disk logging but still no change.
Logs are taken during the time of issue.
CLI Commands:
exec log filter category 0
0: memory 1: disk 2: fortianalyzer 3: forticloud
execute log filter device 0 execute log filter view-lines 100 exec log filter dump category: traffic device: memory start-line: 1 view-lines: 100 max-checklines: 0 HA member: log search mode: on-demand pre-fetch-pages: 2 Oftp search string:
exec log display
0 logs found. 0 logs returned. 2.0% of logs has been searched.
Apply the miglogd debug:
dia debug application miglogd -1
dia debug enable Files to be searched: file_no=60704, start line=0, end_line=4962 file_no=60705, start line=0, end_line=14630 file_no=60706, start line=0, end_line=15680 file_no=60707, start line=0, end_line=15997 file_no=60708, start line=0, end_line=16536 file_no=60709, start line=0, end_line=16917 file_no=60710, start line=0, end_line=17024 file_no=60711, start line=0, end_line=16836 file_no=60712, start line=0, end_line=16704 file_no=60713, start line=0, end_line=16670 file_no=60714, start line=0, end_line=16572 file_no=60715, start line=0, end_line=16487 file_no=60716, start line=0, end_line=16237 session ID=20, total logs=201265 <- write_file_line_nos_into_index_file:359 Failed to open search index file. session_ID=20 <- back ground search. process ID=24530, session_id=20 start line=1 view line=100 pre-fetch-pages=2
on-demand back ground search exit. process ID=24530, session_id=20, status=process_on-demand_pending 0 logs found. 0 logs returned. 2.0% of logs has been searched.
<24351> miglog_file_open()-348: Error creating/opening /var/log/log/root/tlog <24351> miglog_write_local_log()-1690: Error opening log file /var/log/log/root/tlog <24351> __process_flush_cache()-581: Request flush local log caches. vd=0, cat=0, device=0
file_no=60716, start line=0, end_line=16237 session ID=21, total logs=201679 write_file_line_nos_into_index_file:359 Failed to open search index file. session_ID=21 >>>> error failed to open search index. back ground search. process ID=24535, session_id=21 start line=1 view line=500 pre-fetch-pages=2 ( subtype "forward" ) on-demand back ground search exit. process ID=24535, session_id=21,status=process_on-demand_pending >> Unable to open the process getting stuck <24351> miglog_file_open()-348: Error creating/opening /var/log/log/root/tlog <-
After seeing the error in the index file, it tried to kill the httpsd and miglogd, but the same issue was still faced. If the issue still persists, refer to the below steps.
When searching logs stored in disk or memory, the log search daemon (log_se)'s child process (chile process) creates a temporary index file on the local hard disk. If this process fails, the log search will be stopped.
Since this cannot be solved in terms of S/W, it seems that the following method can be performed.
- Reboot (there is a possibility that the damaged disk can be recovered by rebooting).
- When searching logs after method 1, a log such as 'write_file_line_nos_into_index_file:360 Failed to open search index file'. session_ID=x' is recorded in the miglogd log. If this happens, there may be a way to format the hard disk.
NOTE: After disabling the hard disk logging and using the memory alone will also have such a problem.
This is because when doing any kind of log search, it does not matter if it is from a disk log or memory log, the log search child process will make a temporary index file on disk and if that step fails, the log search will die too. That is exactly what is shown in the debug log.
Related document:
Log-related diagnostic commands
|