|
Scenario:
- The customer was getting an error at 2% (loading too slow) in GUI while checking for the forward traffic logs in GUI.
- Changed the status to memory but still no logs same error.
- Disabled the disk logging but still no change.
Logs are taken during the time of issue.
CLI Commands:
execute log filter category 0 <----- Traffic category.
Choose the appropriate device from the list of available devices:
execute log filter device
Available devices:
0: memory
1: disk
2: fortianalyzer
3: fortianalyzer-cloud
4: forticloud
execute log filter view-lines 100
execute log filter dump
category: traffic
device: memory
start-line: 1
view-lines: 100
max-checklines: 0
HA member:
log search mode: on-demand
pre-fetch-pages: 2
Oftp search string:
execute log display
0 logs found.
0 logs returned.
2.0% of logs has been searched.
Apply the miglogd debug:
diagnose debug application miglogd -1
diagnose debug enable
Files to be searched:
file_no=60704, start line=0, end_line=4962
file_no=60705, start line=0, end_line=14630
file_no=60706, start line=0, end_line=15680
file_no=60707, start line=0, end_line=15997
file_no=60708, start line=0, end_line=16536
file_no=60709, start line=0, end_line=16917
file_no=60710, start line=0, end_line=17024
file_no=60711, start line=0, end_line=16836
file_no=60712, start line=0, end_line=16704
file_no=60713, start line=0, end_line=16670
file_no=60714, start line=0, end_line=16572
file_no=60715, start line=0, end_line=16487
file_no=60716, start line=0, end_line=16237
session ID=20, total logs=201265 <-----
write_file_line_nos_into_index_file:359 Failed to open search index file. session_ID=20 <-----
back ground search. process ID=24530, session_id=20
start line=1 view line=100 pre-fetch-pages=2
on-demand back ground search exit. process ID=24530, session_id=20, status=process_on-demand_pending
0 logs found.
0 logs returned.
2.0% of logs has been searched.
<24351> miglog_file_open()-348: Error creating/opening /var/log/log/root/tlog
<24351> miglog_write_local_log()-1690: Error opening log file /var/log/log/root/tlog
<24351> __process_flush_cache()-581: Request flush local log caches. vd=0, cat=0, device=0
file_no=60716, start line=0, end_line=16237
session ID=21, total logs=201679
write_file_line_nos_into_index_file:359 Failed to open search index file. session_ID=21 >>>> error failed to open search index.
back ground search. process ID=24535, session_id=21
start line=1 view line=500 pre-fetch-pages=2
( subtype "forward" )
on-demand back ground search exit. process ID=24535, session_id=21,status=process_on-demand_pending >> Unable to open the process getting stuck
<24351> miglog_file_open()-348: Error creating/opening /var/log/log/root/tlog <-----
Disable the debug processes with the following commands:
diagnose debug disable
diagnose debug reset
After seeing the error in the index file, try killing the httpsd, and miglogd processes :
fnsysctl killall httpsd
fnsysctl killall miglogd
If the issue still persists, refer to the following steps.
When searching logs stored in disk or memory, the log search daemon (log_se)'s child process (chile process) creates a temporary index file on the local hard disk. If this process fails, the log search will be stopped.
Since this cannot be solved in terms of S/W, it seems that the following method can be performed.
- Reboot (there is a possibility that the damaged disk can be recovered by rebooting).
- When searching logs after method 1, a log such as 'write_file_line_nos_into_index_file:360 Failed to open search index file'. session_ID=x' is recorded in the miglogd log. If this happens, there may be a way to format the hard disk.
Note: After disabling the hard disk logging and using the memory alone will also have such a problem.
This is because when doing any kind of log search, it does not matter if it is from a disk log or memory log, the log search child process will make a temporary index file on disk and if that step fails, the log search will die too. That is exactly what is shown in the debug log.
Related document:
Log-related diagnostic commands
|
