| Solution |
Scenario:
- The customer was getting an error at 2% (loading too slow) in GUI while checking for the forward traffic logs in GUI.
- Changed the status to memory but still no logs same error.
- Disabled the disk logging but still no change.
Logs are taken during the time of issue.
CLI Commands:
# exec log filter category 0
0: memory
1: disk
2: fortianalyzer
3: forticloud
# execute log filter device 0
# execute log filter view-lines 100
# exec log filter dump
category: traffic
device: memory
start-line: 1
view-lines: 100
max-checklines: 0
HA member:
log search mode: on-demand
pre-fetch-pages: 2
Oftp search string:
# exec log display
Files to be searched:
file_no=60704, start line=0, end_line=4962
file_no=60705, start line=0, end_line=14630
file_no=60706, start line=0, end_line=15680
file_no=60707, start line=0, end_line=15997
file_no=60708, start line=0, end_line=16536
file_no=60709, start line=0, end_line=16917
file_no=60710, start line=0, end_line=17024
file_no=60711, start line=0, end_line=16836
file_no=60712, start line=0, end_line=16704
file_no=60713, start line=0, end_line=16670
file_no=60714, start line=0, end_line=16572
file_no=60715, start line=0, end_line=16487
file_no=60716, start line=0, end_line=16237
session ID=20, total logs=201265 <-
write_file_line_nos_into_index_file:359 Failed to open search index file. session_ID=20 <-
back ground search. process ID=24530, session_id=20
start line=1 view line=100 pre-fetch-pages=2
on-demand back ground search exit. process ID=24530, session_id=20, status=process_on-demand_pending
0 logs found.
0 logs returned.
2.0% of logs has been searched.
<24351> miglog_file_open()-348: Error creating/opening /var/log/log/root/tlog
<24351> miglog_write_local_log()-1690: Error opening log file /var/log/log/root/tlog
<24351> __process_flush_cache()-581: Request flush local log caches. vd=0, cat=0, device=0
file_no=60716, start line=0, end_line=16237
session ID=21, total logs=201679
write_file_line_nos_into_index_file:359 Failed to open search index file. session_ID=21 >>>> error failed to open search index.
back ground search. process ID=24535, session_id=21
start line=1 view line=500 pre-fetch-pages=2
( subtype "forward" )
on-demand back ground search exit. process ID=24535, session_id=21,status=process_on-demand_pending >> Unable to open the process getting stuck
<24351> miglog_file_open()-348: Error creating/opening /var/log/log/root/tlog <-
After seeing the error in the index file, it was tried to kill the httpsd and miglogd but still, the same issue was faced. If the issue kindly is still persisting, refer to the below steps.
When searching logs stored in disk or memory, the log search daemon (log_se)'s child process (chile process) creates a temporary index file on the local hard disk. If this process fails, the log search will be stopped.
Since this cannot be solved in terms of S/W, it seems that the following method can be performed.
1) Reboot (there is a possibility that the damaged disk can be recovered by rebooting).
2) When searching logs after method 1, a log such as 'write_file_line_nos_into_index_file:360 Failed to open search index file. session_ID=x' is recorded in the miglogd log. If this happens, there may be a way to format the hard disk.
NOTE: After disabling the hard disk logging and using the memory alone will also have such a problem.
This is because when doing any kind of log search, it does not matter if it is from a disk log or memory log, the log search child process will make a temporary index file on disk and if that step fails, the log search will fail too. That is exactly what is shown in the debug log.
Related document:
https://docs.fortinet.com/document/fortigate/7.2.4/administration-guide/668197/log-related-diagnose-...
|
