Configuration Example: FortiGate remote monitoring and logging CLI command output into a file

[jskrivan_FTNT]

Description

Monitoring a FortiGate unit remotely, and logging text outputs of diagnostic CLI commands to a local file, can be used in conjunction with SNMP to investigate the status of a FortiGate unit. The technique described in this document is useful for performance testing and/or troubleshooting. This method logs into a FortiGate unit, runs specific FortiGate CLI commands, and logs their outputs to a text file.
 
The requirements include following:

 
•  An operating system that supports shell scripting: Linux, Unix, OS X.
•  The target FortiGate unit must have an SSH access enabled.
 
This example is performed on a FortiGate 5101B (FortiOS firmware version 5.0.6-build0271), but any other FortiGate model is likely to work.
 

Scope

FortiOS firmware - all supported versions

Solution

Configuration Procedure
 
The configuration procedure includes the following steps:

 
•  Create the shell script and use FortiGate CLI commands.
•  Execute the shell script to a FortiGate unit, and log the output to a file.
 
Before beginning the creation procedure, it is important to understand the directory structure that is being used in this document:
 
/FortiGate5101C          <- This is where output log files are stored.
/FortiGate5101C/Script   <- This is where scripts are stored.
 

 
Step 1 – Create the Shell Script
 
The first step is to create a shell script that includes all FortiGate CLI commands. Let’s call the script script-fg-spm.sh.
 
Starting Directory: /FortiGate5101C/Script
 
Example Shell Script:
 
#! /bin/bash
 
        echo exec date
        sleep 1
        echo exec time
        sleep 1
        echo get sys status
        sleep 1
        echo get system performance status
        sleep 1
        echo diag npu spm status 0
        sleep 1
        echo diag sys session stat \| grep count
        sleep 1
        echo diag sys session6 stat \| grep count
        sleep 1
        echo diag sys top 1 10
        sleep 4
        echo q
        sleep 1
        echo diag debug crash read
        sleep 1
        echo diag debug crash clear
        sleep 1
 
#End of Script
 
Dictionary for the above shell script:
 
sleep x: wait for x number of second(s) before moving on to the next command.
echo q: quite
echo diag debug crash read: reads the crash log
echo diag debug crash clear: clears the crash log
 The intentions of the crash commands are to display only crash messages for a given time frame.
 
Once the script has been created, change permissions of the file to allow its execution:
 
chmod 755 script-fg-spm.sh
 

Step 2 – Execute the Shell Script
 
Starting Directory: /FortiGate5101C
 
The script execution examples below requires a password and host check user interaction.
 
Without VDOMs:
 
(sleep 1; while [ 1 ]; do ./Script/script-fg-spm.sh ; done) | ssh -t -t admin@10.100.23.40 > FG5101C-Monitor.txt
 
With VDOMs:
 
(echo config global; sleep 1; while [ 1 ]; do ./Script/script-fg-spm.sh ; done) | ssh -t -t admin@10.100.23.40 > FG5101C-Monitor.txt
 

Install sshpass to execute the script in a non-interactive mode without entering the password every time.
 
How to install sshpass on the MAC OS X Mavericks:
 
• Download sshpass-1.05, open the directory in a terminal and run the command ./configure.
• An installation window will appear, select Get Xcode and install it.
• Open Xcode and walk through the acceptable use policy.
• Run the command ./configure again.
• Rename the sshpass-1.05 directory to sshpass, copy it to the /Applications Directory.
• Edit your .bash_profile in your home directory (users/name) and add the following line.
     o export PATH="/Applications/sshpass":$PATH
• Once the path has been added, type export PATH="/Applications/sshpass":$PATH in the terminal.
• To verify if sshpass works type sshpass –V in the terminal to display the version number.
 
How to install sshpass on Linux OS:
 
sudo apt-get install sshpass
 

 
The script execution examples below are non-interactive. The ssh-pass –p command passes credentials to SSH.
 
Without VDOMs:
 
(sleep 1; while [ 1 ]; do ./Script/script-fg-spm.sh ; done) | sshpass -p 'fortinet' ssh -t -t -o StrictHostKeyChecking=no admin@10.100.23.40 > FG5101C-Monitor.txt
 
With VDOMs:
 
(echo config global; sleep 1; while [ 1 ]; do ./Script/script-fg-spm.sh ; done) | sshpass -p 'fortinet' ssh -t -t -o StrictHostKeyChecking=no admin@10.100.23.40 > FG5101C-Monitor.txt
 
Running the script using the lines above will result in the outputs of the FortiGate CLI commands being logged to a file name FG5101C-Monitor.txt
  

 
Step 3 – Viewing the Running Output of the Script
 
Start Directory: /FortiGate5101C
 
To view the current output of the script that is being run, use the following command.
 
tail –f FG5101C-Monitor.txt
 

 
Example output:
 
c01s03 (global) # diag debug cr re^M
 
c01s03 (global) # diag debug cr cl^M
 
c01s03 (global) # exec date^M
current date is: 2014-03-19
 
c01s03 (global) # exec time^M
current time is: 10:15:39
last ntp sync: never
 
c01s03 (global) # get sys status^M
Version: FortiGate-5101C v5.0,build0271,140124 (GA Patch 6)
Virus-DB: 16.00560(2012-10-19 08:31)
Extended DB: 1.00000(2012-10-17 15:46)
Extreme DB: 1.00000(2012-10-17 15:47)
IPS-DB: 4.00345(2013-05-23 00:39)
IPS-ETDB: 0.00000(2001-01-01 00:00)
Serial-Number: FG-51C3E12800155
Botnet DB: 1.00000(2012-05-28 22:51)
BIOS version: 04000008
System Part-Number: P10520-03
Log hard disk: Available
Hostname: c01s03
Operation Mode: NAT
Current virtual domain: elbc-mgmt
Max number of virtual domains: 10
Virtual domains status: 2 in NAT mode, 0 in TP mode
Virtual domain configuration: enable
FIPS-CC mode: disable
Current HA mode: standalone
Config-Sync: Master
Branch point: 271
Release Version Information: GA Patch 6
FortiOS x86-64: Yes
System time: Wed Mar 19 10:15:40 2014
 
c01s03 (global) # get system performance status^M
CPU states: 11% user 52% system 0% nice 37% idle
CPU0 states: 20% user 34% system 0% nice 46% idle
CPU1 states: 9% user 58% system 0% nice 33% idle
CPU2 states: 10% user 60% system 0% nice 30% idle
CPU3 states: 12% user 51% system 0% nice 37% idle
CPU4 states: 9% user 46% system 0% nice 45% idle
CPU5 states: 8% user 56% system 0% nice 36% idle
CPU6 states: 10% user 51% system 0% nice 39% idle
CPU7 states: 7% user 58% system 0% nice 35% idle
Memory states: 13% used
Average network usage: 250549 kbps in 1 minute, 15136715 kbps in 10 minutes, 35713703 kbps in 30 minutes
Average sessions: 6844 sessions in 1 minute, 36666 sessions in 10 minutes, 45569 sessions in 30 minutes
Average session setup rate: 3266 sessions per second in last 1 minute, 5698 sessions per second in last 10 minutes, 13771 sessions per second in last 30 minutes
Virus caught: 0 total in 1 minute
IPS attacks blocked: 0 total in 1 minute
Uptime: 4 days,  15 hours,  7 minutes
 
c01s03 (global) # diag npu spm status 0^M
XH0
Onboard             : sp_0
Version             : 1.317 (Dec 18 2013 16:40:31)
 

IPS Version                   : 2.179
Build Time                    : 01/15/2014 12:00
IPS rule version              : 4.345
AV rule version               : 10.974
 
IPS status          : STANDBY
IPS mask            : 00000008
Serial Number       : __FAKE__SN__
Bootloader          : v0001-r26758
Module version      : 5101C-r1
CPU                 : XLP432 Rev B0
 
CPU utilization     : 4%
  cpu 00 - 03    0% 80% 0% 0%
  cpu 04 - 07    3% 4% 3% 0%
  cpu 08 - 11    0% 3% 1% 3%
  cpu 12 - 15    9% 6% 4% 4%
  cpu 16 - 19    6% 4% 3% 6%
  cpu 20 - 23    12% 9% 7% 4%
  cpu 24 - 27    3% 1% 4% 4%
  cpu 28 - 31    4% 1% 4% 3%
 
Memory utilization  : 19%
Second on SP        : 400003:400008
Loops on monitor    : 400853632:400858543
 

Master                        Onboard, sp_0
Number of Slave Module        0
Switch port                   6 7 8 9
 

c01s03 (global) # diag sys session stat | grep count^M
misc info:       session_count=79696 setup_rate=36942 exp_count=0 clash=0
fqdn_count=00000000
 
c01s03 (global) # diag sys session6 stat | grep count^M
misc info:       session_count=0 setup_rate=0 exp_count=0 clash=0
 
c01s03 (global) # diag sys top 1 10^M
Run Time:  4 days, 15 hours and 7 minutes
0U, 0S, 100I; 11975T, 10354F, 512KF
     sessionsync      616      R      48.5     0.0
     sessionsync      614      R      44.6     0.0
     sessionsync      608      R      42.7     0.0
     sessionsync      615      R      41.7     0.0
     sessionsync      613      R      37.8     0.0
     sessionsync      612      R      37.8     0.0
     sessionsync      610      R      36.8     0.0
     sessionsync      609      R      36.8     0.0
     sessionsync      611      R      34.9     0.0
     sessionsync      606      R      34.9     0.0
Run Time:  4 days, 15 hours and 7 minutes
19U, 70S, 11I; 11975T, 10346F, 512KF
     sessionsync      616      R      65.0     0.0
     sessionsync      614      R      55.3     0.0
     sessionsync      615      R      53.3     0.0
     sessionsync      612      R      47.5     0.0
     sessionsync      613      S      45.6     0.0
     sessionsync      609      S      42.7     0.0
     sessionsync      611      S      41.7     0.0
     sessionsync      607      R      41.7     0.0
     sessionsync      610      R      40.7     0.0
     sessionsync      608      R      40.7     0.0
Run Time:  4 days, 15 hours and 7 minutes
14U, 73S, 13I; 11975T, 10338F, 512KF
     sessionsync      615      R      68.3     0.0
     sessionsync      616      R      62.3     0.0
     sessionsync      614      R      48.5     0.0
     sessionsync      613      R      45.5     0.0
     sessionsync      610      R      45.5     0.0
     sessionsync      609      R      45.5     0.0
     sessionsync      612      R      41.5     0.0
     sessionsync      611      R      40.5     0.0
     sessionsync      605      R      40.5     0.0
     sessionsync      608      R      39.6     0.0
 
c01s03 (global) # ^M
c01s03 (global) # diag debug cr re^M
 
c01s03 (global) # diag debug cr cl^M
 
To stop the script, use the keyboard command Ctrl+C.