FortiGate
FortiGate Next Generation Firewall utilizes purpose-built security processors and threat intelligence security services from FortiGuard labs to deliver top-rated protection and high performance, including encrypted traffic.
asengar
Staff
Staff
Article Id 252194
Description

This article describes the issue when the customer is unable to see the forward traffic logs either in memory or disk or another remote logging device.

Scope FortiOS.
Solution

Scenario:

 

  • The customer was getting an error at 2% (loading too slow) in GUI while checking for the forward traffic logs in GUI.
  • Changed the status to memory but still no logs same error.
  • Disabled the disk logging but still no change.

 

Logs are taken during the time of issue.

 

CLI Commands:

 

exec log filter category 0

0: memory
1: disk
2: fortianalyzer
3: forticloud


execute log filter device 0
execute log filter view-lines 100
exec log filter dump
category: traffic
device: memory
start-line: 1
view-lines: 100
max-checklines: 0
HA member:
log search mode: on-demand
pre-fetch-pages: 2
Oftp search string:

 

exec log display

0 logs found.
0 logs returned.
2.0% of logs has been searched.

 

Apply the miglogd debug:

dia debug application miglogd -1

dia debug enable
Files to be searched:
file_no=60704, start line=0, end_line=4962
file_no=60705, start line=0, end_line=14630
file_no=60706, start line=0, end_line=15680
file_no=60707, start line=0, end_line=15997
file_no=60708, start line=0, end_line=16536
file_no=60709, start line=0, end_line=16917
file_no=60710, start line=0, end_line=17024
file_no=60711, start line=0, end_line=16836
file_no=60712, start line=0, end_line=16704
file_no=60713, start line=0, end_line=16670
file_no=60714, start line=0, end_line=16572
file_no=60715, start line=0, end_line=16487
file_no=60716, start line=0, end_line=16237
session ID=20, total logs=201265  <-
write_file_line_nos_into_index_file:359 Failed to open search index file. session_ID=20 
<-
back ground search. process ID=24530, session_id=20

start line=1 view line=100 pre-fetch-pages=2

on-demand back ground search exit. process ID=24530, session_id=20, status=process_on-demand_pending
0 logs found.
0 logs returned.
2.0% of logs has been searched.


<24351> miglog_file_open()-348: Error creating/opening /var/log/log/root/tlog

<24351> miglog_write_local_log()-1690: Error opening log file /var/log/log/root/tlog
<24351> __process_flush_cache()-581: Request flush local log caches. vd=0, cat=0, device=0

file_no=60716, start line=0, end_line=16237
session ID=21, total logs=201679
write_file_line_nos_into_index_file:359 Failed to open search index file. session_ID=21 >>>> error failed to open search index.
back ground search. process ID=24535, session_id=21
start line=1 view line=500 pre-fetch-pages=2
( subtype "forward" )
on-demand back ground search exit. process ID=24535, session_id=21,status=process_on-demand_pending >> Unable to open the process getting stuck
<24351> miglog_file_open()-348: Error creating/opening /var/log/log/root/tlog  <-

 

After seeing the error in the index file, it tried to kill the httpsd and miglogd, but the same issue was still faced. If the issue still persists, refer to the below steps.

 

When searching logs stored in disk or memory, the log search daemon (log_se)'s child process (chile process) creates a temporary index file on the local hard disk. If this process fails, the log search will be stopped.

Since this cannot be solved in terms of S/W, it seems that the following method can be performed.

 

  1. Reboot (there is a possibility that the damaged disk can be recovered by rebooting).

  2. When searching logs after method 1, a log such as 'write_file_line_nos_into_index_file:360 Failed to open search index file'. session_ID=x' is recorded in the miglogd log. If this happens, there may be a way to format the hard disk.

 

NOTE: After disabling the hard disk logging and using the memory alone will also have such a problem.

 

This is because when doing any kind of log search, it does not matter if it is from a disk log or memory log, the log search child process will make a temporary index file on disk and if that step fails, the log search will die too. That is exactly what is shown in the debug log.

 

Related document:

Log-related diagnostic commands