Technical Tip: FortiGate monitoring script

[Jackie_T]

Description

This article describes how to run a script remotely on a  FortiGate, using Tera Term software to capture the data on a timely basis. This is useful to collect info to analyze the overall health of the device performance and it is also used to capture intermittent issue which occurs randomly such as CPU or memory spike.

Scope

FortiGate.

Solution

Refer to the following KB article for a suggested diagnostic script: Technical Tip: TAC debug script with TeraTerm. Many SSH tools can be used, but in this example, Teraterm will be used to run the monitoring script.

Here are the steps to use the monitoring script with Teraterm:

To run the script follow the steps mentioned below.

1. Download TeraTerm.
   
2. Run ttermpro.exe from a PC connected to the LAN or by console and log in to the firewall.
3. Logging in to the FortiGate:
   1. If connecting by SSH, select TCP/IP -> Service -> SSH and enter the Host IP address.  If this is the first time connecting, select Continue to accept the SSH key. Enter the admin username and password credentials and select OK.
   2. If connecting to the console, select Serial and the active COM port from the drop-down menu. Enter the admin username and password credentials.
4. To start logging, select File -> Log and then choose where to save the log file.
5. Run the script by selecting Control -> Macro and selecting the script (file with .ttl extension).
6. Do not close the logging or macro window.

Change the terminal width settings as shown below:

1. Go to Setup then Terminal.
2. Change the settings (Terminal Size) to as shown below.

Figure 1.

Note:  

• The script must be in .ttl format otherwise the macro script tool will not execute some commands.
• Different scenarios can require focusing on a specific daemon by collecting command output designated for that daemon. Find attached an example of ttl scrip to troubleshoot high memory usage by WAD processes - filename 'wad_mon_new.ttl'.
• Another scenario can be troubleshooting intermittent traffic drops. For sniffer-based script can be used such as the one described in the article: Technical Note: Using Tera Term sniffer with a FortiGate to save output to different files every <N>...
• If the script gets logout automatically, adjust the login timeout period following this command:

config system global
    set admin-ssh-grace-time <number_of_seconds> << max 3600 seconds

    set admintimeout  <number_of_minutes< << max 480 minutes
end 

Packet captures are usually not run using scripts:

For a long-running packet capture, it is not necessary to use a script.

Instead open an ssh client, log output to a file, and run a packet sniffer on the command line. See links below.

Relevant links for packet capture:

FortiOS v7.4.8 Administration Guide: Performing a sniffer trace or packet capture 

Technical Tip: How to create a log file of a session using PuTTY
Technical Tip: Logging FortiGate administrative SSH sessions from Linux/MacOS endpoints

Increasing Number of Lines in terminal window:

If logging to a file, the log stores all output of the monitoring script including previous lines no longer visible in the terminal window. If necessary, the number of lines of scrollback can also be increased by navigating to Setup > Window > Scroll buffer. The scroll buffer is set to 10,000 lines by default, and can be increased up to 500,000 lines. See the third-party reference TeraTerm maximum scroll buffer size.

Note: Tera Term consumes <scroll buffer size> * <terminal width> * 5 bytes of memory for the scroll buffer.

Related article:
TTL command reference