Technical Tip: Transferring historical logs from a FortiGate hard disk to a FortiAnalyzer

[mricardez]

Description

This article describes the steps required to move logs previously stored on a FortiGate Hard Disk to a FortiAnalyzer so that those logs can be included in FortiView or Reports. 

An example of this might be purchasing a FortiAnalyzer after a FortiGate has been in production.

It describes using an open-source tool called  lz4_reader on a Windows workstation. 

Notes:

1. It is possible to use the same tool on a MAC or Linux workstation but need to choose the -jar option when running the executable and need to have JDK (Java Development Kit) installed.
2. If downloading a log file from the FortiOS GUI, it will not be compressed in LZ4 format, thus bypassing the need to perform the conversion described in this tech note.
3. Refer to 'Technical Note: Importing multiple logs into FortiAnalyzer' in the Related Articles on how to inject them all back in FortiAnalyzer if needed as one single file.

Scope

FortiAnalyzer.

Solution

The logs stored on the FortiGate Hard Disk are in format  LZ4 and can not be directly imported to the FortiAnalyzer without first making some modifications. 

It is necessary to translate the LZ4 logs files to txt format using a FortiGate tool called 'lz4_reader'.

Note: The tool is attached to this KB article for the convenience of readers. It is provided 'as is' and is not maintained by Fortinet.

1. Export all logs from the FortiGate Hard Disk to the FTP server.

FGTXXXXXXXXXX034 (root) # execute backup disk alllogs ftp 192.168.10.100 ftptest ftptest

Please wait...

Connect to ftp server 192.168.10.100 ...

Sent log file tlog.65147 to ftp server as tlog_FGTXXXXXXXXXX034_root_20170421_020000 OK.

Please wait...

Connect to ftp server 192.168.10.100 ...

Sent log file elog.65129 to ftp server as elog_FGTXXXXXXXXXX034_root_20170421_020000 OK.

Please wait...

Connect to ftp server 192.168.10.100 ...

Sent log file plog.65438 to ftp server as plog_FGTXXXXXXXXXX034_root_20170421_001645 OK.

Please wait...

Connect to ftp server 192.168.10.100 ...

Sent log file rlog.65147 to ftp server as rlog_FGTXXXXXXXXXX034_root_20170421_020000 OK.

Please wait...

FGTXXXXXXXXXX034 (root) #

2. Uncompress the "lz4_reader” log conversion tool.
   

   Uncompress (using a tool like WinRAR) 'lz4_reader' (a 3rd party tool attached to this technote for convenience) into a path on a local PC.

   In the example below, the path used is 'C:\Users\MARK\Documents\lza_reader>'.
   
   

   Note: The 'lz4_reader' tools translate LZ4 logs to TXT format.  In the example outlined in this article, the tool was run in Windows 10 with Java v8 ( build 1.8.0_77-b03).
   
   

   C:\Users\MARK\Documents\lza_reader>dir

   El volumen de la unidad C es Windows

   El número de serie del volumen es: 641A-5B1F

   Directorio de C:\Users\MARK\Documents\lza_reader

   27/04/2017 03:01 p. m. <DIR> .

   27/04/2017 03:01 p. m. <DIR> ..

   11/10/2016 12:48 p. m. 6,148 .DS_Store

   11/10/2016 12:49 p. m. 4,096 ._.DS_Store

   11/10/2016 12:47 p. m. 3,253,658 log_reader.jar

   29/09/2016 01:27 p. m. 693 run.bat

   4 archivos 3,264,595 bytes

   2 dirs 1,701,749,608,448 bytes libres

   C:\Users\MARK\Documents\lza_reader>

3. Translate the LZ4 file into TXT format.
   In a CMD of Windows run the command 'run' into the directory where it was uncompressed.
   Choose the option 1.
   Type the complete FG log file path in your Windows PC
   The tool 'lz4_reader' will create a directory and will put all files changed to TXT into this path.

   C:\Users\MARK\Documents\lza_reader>run

   Please input command number and enter...

   To read a log, enter 1

   To terminate the reader, enter 2

   1

   Input the path of the log you want to read...

   C:\Users\MARK\Downloads\tlog_FGTXXXXXXXXXX034_root_20170421_020000

   The path you input is C:\Users\MARK\Downloads\tlog_FGTXXXXXXXXXX034_root_20170421_020000

   All readable contents are saved to C:\Users\MARK\Downloads\tlog_FGTXXXXXXXXXX034_root_20170421_020000_readable.

   Presione una tecla para continuar . . .

   Please input command number and enter...

   To read a log, enter 1

   To terminate the reader, enter 2

   2

4. Rename the file extension from 'txt' to 'log'.
   
   

   C:\Users\MARK\Downloads\tlog_FGTXXXXXXXXXX034_root_20170421_020000_readable> dir

   El volumen de la unidad C es Windows

   El número de serie del volumen es: 641A-5B1F

   Directorio de C:\Users\MARK\Downloads\tlog_FGTXXXXXXXXXX034_root_20170421_020000_readable

   27/04/2017 03:05 p. m. <DIR> .

   27/04/2017 03:05 p. m. <DIR> ..

   27/04/2017 02:59 p. m. 3,680,094 tlog_FGTXXXXXXXXXX034_root_20170421_020000

   27/04/2017 03:05 p. m. 35,075,188 tlog_FGTXXXXXXXXXX034_root_20170421_020000_readable.txt

   2 archivos 38,755,282 bytes

   2 dirs 1,701,587,505,152 bytes libres

   C:\Users\MARK\Downloads\tlog_FGTXXXXXXXXXX034_root_20170421_020000_readable>

   C:\Users\MARK\Downloads\tlog_FGTXXXXXXXXXX034_root_20170421_020000_readable>

   C:\Users\MARK\Downloads\tlog_FGTXXXXXXXXXX034_root_20170421_020000_readable> --> Rename tlog_FGTXXXXXXXXXX034_root_20170421_020000_readable.txt.
   tlog_FGTXXXXXXXXXX034_root_20170421_020000_readable.log

   C:\Users\MARK\Downloads\tlog_FGTXXXXXXXXXX034_root_20170421_020000_readable> dir

   El volumen de la unidad C es Windows

   El número de serie del volumen es: 641A-5B1F

   Directorio de C:\Users\MARK\Downloads\tlog_FGTXXXXXXXXXX034_root_20170421_020000_readable

   27/04/2017 03:09 p. m. <DIR> .

   27/04/2017 03:09 p. m. <DIR> ..

   27/04/2017 02:59 p. m. 3,680,094 tlog_FGTXXXXXXXXXX034_root_20170421_020000

   27/04/2017 03:05 p. m. 35,075,188 tlog_FGTXXXXXXXXXX034_root_20170421_020000_readable.log

   2 archivos 38,755,282 bytes

   2 dirs 1,701,659,672,576 bytes libres

   C:\Users\MARK\Downloads\tlog_FGTXXXXXXXXXX034_root_20170421_020000_readable>
   

   Note: With recent format changes the file should be renamed with the below format before it can be imported:
   
   '[Firewall_Serial_Number].[VdomName].[tlog].[Date].[Timestamp].log'.
   
   For instance: 'FGTXXXXXXXXXX034.root.tlog.20170421_020000'.

5. From the FortiAnalyzer CLI, import the txt file with extension .log by FTP.

   FAZVM64 # execute log import ftp 192.168.10.100 ftptest ftptest tlog_FGTXXXXXXXXXX034_root_20170421_020000_readable.log FGTXXXXXXXXXX034

   Do you want to continue? (y/n)y

   Log Import Info: Connect to ftp server 192.168.10.100 ...

   Log Import Info: Found 1 .log or .csv files in remote folder : tlog_FGTXXXXXXXXXX034_root_20170421_020000_readable.log .

   Log Import Info: 1 log files found in remote folder, MAX import file setting is 10000, so 1 files will be imported.

   Log Import Info: Downloading files from 192.168.10.100 ...#

   Log Import Info: Log file tlog_FGTXXXXXXXXXX034_root_20170421_020000_readable.log was successfully imported to FGTXXXXXXXXXX034/tlog.1492668005.log.

   Log Import Info: 1 log files are imported.

   Log Import Info:

   1 files are processed, 0 files remain.

   FAZVM64 #

Once the FortiAnalyzer has finished importing the logs into the SQL database, the logs will be visible in LogView and FortiView, and available during report generation.

Related article:

Technical Note: Importing multiple logs into FortiAnalyzer