Native and Remote shutdown of managed FortiSwitch? (without SSHing into it from FortiGate)

[Damien1]

Is there any way to properly shutdown a FortiGate-manged FortiSwitch without having to ssh into it from the FortiGate CLI? 

Also, how about doing so remotely with FortiCloud?

(There's a way to reboot FortiSwitches from the FortiGate GUI and CLI, but not easily from FortiGate Cloud and not at all from FortiExplorer iOS)

The best thing I have been able to come up with is not very convient for remote use and it only sort of works:

1.) On the FortiGate define a custom-command to send to each of my 2 FortiSwitches:

config switch-controller custom-command
    edit "shutdown-sw-custom"
        set description "shutdown now script"
        set command "execute shutdown"
    next
end

2.) Invoke the commands from the ForiGate CLI, or from FortiGate Cloud define and deploy a script containing the commands:

execute switch-controller custom-command shutdown-sw-custom <FortiSwitchF124SerialNumber>
execute switch-controller custom-command shutdown-sw-custom <FortiSwitchF108ESerialNumber>

If it fully worked perfectly/reliably this would be an awkward but still usable workaround for a lack of a scriptable direct command for automation purposes, and also to a 'Shutdown FortiSwitch' button in the FortiGate GUI, FortiGate Cloud, and in FortiExplorer for manually invoked purposes... but when running the defined c ommand with my FortiSwitch F124 as the target I always get

Command fail. Return code -7621

response which halts the execution of a script...even though on the bright side anecdotally (small sample size) the switch shuts down anyway after a few minutes and/or after the FortiGate is shut down.

The command seems to work reliably for shutting down the FortiSwitch 108E FPOE without getting any error messages though.

Is there any way to shut down a managed FortiSwitch without having to SSH into it from the FortiGate in that is turn logged into from a computer locally or enabling remote access ?

Surely I'm overlooking a more simple way of shutting down a FortiSwitch manually and in an automation process?

Thanks!

(

Motivation:

Being able to shutdown FortiSwitches before shutting down the FortiGate 60F via automated script and manually in the event of a power outage and a monitoring device detects that the UPS' battery is getting low and other situtations...

command script triggered by the ForiGate shutdown process commencing, webhook from the UPS monitoring device

Running FortiOS 7.0.3 on all components.

)

[Anthony_E]

Hello Damien,

May I propose you to have a look in our Knowledge Base?

You can make research via the search bar, selecting Knowledge base.

https://community.fortinet.com/t5/Knowledge-Base/ct-p/knowledgebase

If you do not find the solution there, somebody will help you.

Regards,

[Damien1]

>>May I propose you to have a look in our Knowledge Base?

>>You can make research via the search bar, selecting Knowledge base.

No doubt people often want to skip reading the documentation and spending the time exploring existing resources (and thereby learning in general in the process) that people have put a lot of work into creating. Prior to this posting I did rather carefully go through literally all of the available documentation for FortiGate units and FortiSwitch units, as well as FortiCloud-related, and I spent time in the Knowledge Base over a period of a few days in addition to the forum. Also checked the FortiNet sub-Reddit, FortiNet YouTube channel. Googled in general.

In doing so, I found some bits of info in both the Knowledge Base and in the Forum of what others have shared that are partially related to some aspects of what I mentioned that I was trying to accomplish and in what manner. But I did not find enough information to the extent that fully addressed what I hoped to accomplish overall, which motivated the time I put into composing my first post here and provide as much info as possible about what I have tried. 

(Also did not find anything at all in the resources that happened to matched the problems I ran into with the same error code response.)

That being said, it is entirely possible that I overlooked some information somewhere. Asking here after having exhausted all available resources, basically. 

If I missed something, I definitely would like to know.

Thanks.

[Markus_M]

Hello Damien,

i cannot think of a way of how to do it better without SSH.

But you can remotely execute SSH commands without logging in with username-password on the traditional way. From what I see from you, you might have worked out SSH key connection. When you can log in like that from an SSH key capable node (the UPS is such node maybe), then you can run the commands as such.

cat scripts/fortizoo.sh  
#!/bin/sh
#vars setup
#local:
FGTVAR="/root/scripts/fortigate.var"
#remote:
FSSH="ssh admin@192.168.200.1"

#create cert-import on FGT via FGTVAR
echo "config root" > $FGTVAR
echo "edit root" >> $FGTVAR

echo "execute switch-controller custom-command shutdown-sw-custom <FortiSwitchF124SerialNumber>" >> $FGTVAR
echo "exit" >> $FGTVAR
sleep 1
$FSSH < $FGTVAR
sleep 5
rm $FGTVAR
sleep 1

I use this script to propagate linux "dehydrated" letsencrypt certificates to a FortiMail and a FortiGate and it writes certs to both devices. I tried to adapt it for your case. It would run for a Linux machine as is, Windows mileage may vary.

On Windows you can run commands like this remotely via putty link (=plink.exe), but I lack experience with it (we do have a community article here with plink).

Best regards,

Markus

[Damien1]

UPDATE 2: next reply below with tested ways of achieving objective

UPDATE: in process of finishing more testing that's looking promising in either script-based approach...the fail errors only seem to happen in a interactive mode (CLI prompt and FG Cloud 'deploy script' and the complaining switch is shutting down more quickly. Can not reproduce the previous single automation stitch failure. SSH and CLI scripts seem to not exit with errors as expected. Will update shortly

Thanks for the input here and shell script example too.

--For shutting down locally in an automated manner:

Okay so if I understand correctly, it looks like the 2 options to shut down managed FortiSwitches are to either use custom switch commands or ssh into the switches by first ssh'ing into the FortiGate (or perhaps directly to the switches with some additional set up) 

I've been considering the sort of approach you outline with either a SSH script or 

an automation stitch action to run them in a CLI script on the FortiGate that is triggered on via webhook (LAN-only as well).

As described in the post, the stumbling block that I run into that I think applies to either approach:

Executing the custom command results in an error message code every time when the FortiSwitch F124 is the target (100% of the time when executed in the CLI console on the FortiGate GUI, although the switch shuts down after a while or after the FortiGate is shut down after that. The other switch shuts down immediately without a error code). 

To isolate this problem, for testing I also created 2 temporary time-based automation stitches to execute the custom command in a script for each of the 2 FortiSwitches. I also executed the custom commands by deploying scripts from FortiGate Cloud for a manual shutdown use case.

In either case these separately tested automations fail to shut down either switch  regardless of whether it was deployed from FortiGate Cloud or in the local time-based automation stitch.

UPDATE:  ^^^ Experienced this once and have not been able to reproduce this failure since.

So it seems that returned errors declaring failure to execute the custom switch command when manually invoked would also cause any CLI script using them (with required admin privileges) to be interrupted and fail to run to completion. It seems that a shell script sshing into the FortiGate and invoking the same custom switch command to shutdown the 124F will be interrupted due to this error, and it appears that it will fail on the 108E as well (I will test it to verify though)

Error codes examples: Command fail. Return code -13 or -7621 and others I've seen more since the original post here.

FortiGate Cloud returns error codes for either switch in a deployed script, and the CLI console on the Fortigate GUI returns an error code only for the 124F.

(The SSH approach also seems to require SSHing into the FortiGate while disabling the 'verify server authenticity' as the key on the FortiGate seems to change periodically with either RSA or ssh-ed25519-based keys from what I've read here and elsewhere perhaps after the FortiGate is updated but I'm not sure.)  

If it is more appropriate for me to file a support ticket please let me know and I can mark your solution as the solution. (I'm new to FortiNet)

--For shutting down remotely & manually:

Perhaps adding a shutdown button next to the existing reboot button for a FortiSwitch in FortiExplorer phone app and FortiGate Cloud may be a feature request for consideration if thought to be useful? The existing reboot and the shutdown and reboot buttons for the FortiGate is handy. 

(apologies if there's a feature request section on the website somewhere that I overlooked)

^^^ UPDATE: the automation stitch in the reply to this post takes care of this.

(In my case, I have used a Raspberry Pi Compute 3 up until acquiring the FortiNet components, on a dedicated VLAN with no internet access to monitor the UPS and auto shut down other computers via SSH, and a few other things via MQTT via a bash script when the UPS battery runs low...

I'm not a security expert, but my preference to use a webhook triggering a CLI script on the FortiGate approach is because I've read in the NSA leaks from a few years ago that private SSH keys that are not manually password protected are particularly high value targets in breaches, so an unencrypted private ssh key with admin privileges for the network firewall that is sitting on a device that will occasionally need to be connected to the internet to get updates does not seem ideal to me, but I could be overly concerned or mistaken here. My logic to use a webhook is that if its API key were to compromised, the most that an attacker could do with it is shut everything down. I will probably do the same on other connected computers that I can set up to react to local webhooks)

Thank you again.

[Damien1 · Accepted solution]

FINAL UPDATES/EDITS ADDED in places below with corrections from other parts of the converstaions in this thread:

Okay, I hope this helps someone else too, so I'll give details on what I've best been able to determine: 

Overall, the motivation of the original post is to verify that there is not a native command to shut down a managed FortiSwitch that I was not finding a reference to.
Markus answers that question as likely no (thanks for verifying that).

The workaround in the post using the custom switch commands initially seemed to be unreliable due to the error messages I encounter in running them both in an interactive FortiGate CLI prompt, and via a SSH script. One of the FortiSwitches was also initially to slow to respond to the commands as well, but that got faster with time for some reason.

Through a fair amount of testing by invoking these custom commands in various contexts, it appears that all of the error messages the FortiGate reports in response can be safely ignored (no errors are reported in the FortiGate logs).

The initial automation script failure on the FortiGate has not repeated itself either.

If there are errors going on, the FortiGate appears to be able to process them nonetheless and scripts do not exit in reaction to the errors.

I no longer have the reliability concerns in using the originally described workarounds.

The following is the description of the objective and proposed solutions to what motivated the initial desire to find a native/supported way of shutting down a managed FortiSwitch without ssh'ing into it from a ForiGate:

OBJECTIVE:
The method to shutdown FortiGate-managed FortiSwitches is to log into the FortiGate via GUI or ssh, and then ssh into each FortiSwitch from the FortiGate CLI.
This requires entering a password and local FortiLink ip address corresponding to each switch (admin is prompted to set a password on the first ssh login a FortiSwitch from the FortiGate)
e.g. FortiGate CLI: # ssh admin@169.254.1.x

For a remote and manual shutdown of FortiSwitches, this seems to require enabling remote access on the FortiGate, as only restart buttons presently exist for FortiSwitches in the FortiGate GUI, FortiCloud GUI, and FortiExplorer.

The objective is to reduce the number of steps involved in shutting down FortiSwitches before shutting down the FortiGate in both manual and in automated manners, and to be able to do so remotely and locally.

An example use case is in the event of a power outage or approaching storm on an aging power grid known to have problems during weather events (U.S. power grid is aging). One may wish to be able quickly shut down a FortiGate and connected FortiSwitches at small business or in a home that are connected to a UPS of a limited battery capacity compared to a large UPS one might find in an enterprise Fortinet setup (assumes that the ISP cable modem, etc. is powered by the UPS as well).

One may wish to do so manually and/or in an automated manner when the monitoring device attached to the UPS detects the UPS battery to be in use or is at a low remaining capacity depending on the situation.

All solutions make use of creating the custom command that is applied to each FortiSwitch as shown in original post:

1.) On the FortiGate define a custom-command to send to each FortiSwitch:

config switch-controller custom-command
    edit "shutdown-sw-custom"
    set description "shutdown now script"
    set command "execute shutdown"
    next
end

2.) The commands can be executed on the FortiGate via an Automation Action CLI script with any available trigger (e.g. incoming webhook)
or in an shell script via SSH from an external device in a few different combinations described in the solutions below:

execute switch-controller custom-command shutdown-sw-custom <FortiSwitchF124SerialNumber>
execute switch-controller custom-command shutdown-sw-custom <FortiSwitchF108ESerialNumber>

The particular failure/error messages encountered in using them with my FortiGate and managed FortiSwitches are noise that should be ignored as far as I have been able to determine
(FortiGate 60F, FortiSwitch 108E-FPOE, FortiSwitch 124F, FortiOS 7.0.3 on all)

SOLUTION 1 - UNIVERSAL SOLUTION FOR ALL USE CASES: ANY DESIRED METHOD TO SHUT DOWN A FORTIGATE ALSO SHUTS DOWN ITS MANAGED FORTISWITCHES

Create an Automation Stitch that detects in the logs that the FortiGate has received a shutdown command, and have the FortiGate shutdown its FortiSwitches first.

--Enables one to be able to easily manually shut down the switches and managing FortiGate via FortiExplorer phone app, FortiCloud GUI, FortiGate GUI, ...
--Reduces an ssh shell script or ssh command to a single command (FortiGate shutdown command), as well as being of use with other Automation Stitches
--Is the solution that shuts down the FS's and FG the most quickly as the shutdown commands for each switch are executed in parallel
--The most simple to use

FortiGate CLI:

config system automation-action
    edit "ShutdownSwitchFS1Action"
        set description "runs switch shutdown custom-command target FS 108E"
        set action-type cli-script
        set minimum-interval 0
        set script "execute switch-controller custom-command shutdown-sw-custom <FortiSwitchF108ESerialNumber>"
        set execute-security-fabric disable
        set accprofile "super_admin"
    next
    edit "ShutdownSwitchFS2Action"
        set description "runs switch custom-command target FS 124F"
        set action-type cli-script
        set minimum-interval 0
        set script "execute switch-controller custom-command shutdown-sw-custom <FortiSwitchF124SerialNumber>"
        set execute-security-fabric disable
        set accprofile "super_admin"
    next
end
config system automation-trigger
    edit "OnFortiGateShutdown"
        set description ''
        set trigger-type event-based
        set event-type event-log
        set logid 32200
    next
end
config sys automation-stitch
    edit "ShutdownSwitchesOnFgShutdown"
        set description ''
        set status enable
        set trigger "OnFortiGateShutdown"
        config actions
            edit 1
                set action "ShutdownSwitchFS1Action"
                set delay 0
                set required disable
            next
            edit 2
                set action "ShutdownSwitchFS2Action"
                set delay 0
                set required disable
            next
        end
    next
end

Notes:
--Can use a more restricted admin account type with minimum privileges required for added security LAST UPDATE/EDIT: (yes, to some extent, but there will still be a lot of access granted by it)

--Can combine both actions into one. I chose to use 2 separate actions in case one were to fail and also take advantage of speed by their execution in  parallel processes

USING SOLUTION 1 WITH A DEVICE MONITORING A UPS THAT IS CONNECTED TO A ETHERNET PORT ON THE FORTIGATE

A.) 'Incoming Webhook' Automation Trigger to trigger the FortiGate shutdown:
Possibly the most secure method? If the FortiGate is set up with a public or private CA (i.e. such that no browser warnings about the SSL/TLS connection to the FortiGate GUI is insecure), then the connected client device and FortiGate can verify one another's identity with the security certificates.

LAST UPDATE/EDIT: Thanks @Markus_M for clarifying that this does not pose any issues so crossing this out (got the initial misimpression that the SSH key can periodically change from an older post that no longer applies)
If I understand correctly, the SSH host key on the FortiGate changes periodically, so a shell script would have to be set for the ssh connection to Not verify the identity of the host to guarantee it will not be interrupted with a warning prompt.

OR

B.) Run execute shutdown command on the FortiGate via SSH. Numerous ways to do this, but in line with the method that Markus shows, on the monitoring device.

i) create file forti_shutdown_cmds.txt with the contents:

execute shutdown
y
exit

(probably best to make sure that the new line characters are new line \n only and not Windows \r\n)

ii.) create shell script shutdownfg.sh with contents:

ssh -T -o "StrictHostKeyChecking=no" -p <FG_SSH_PORT> AdminName@FG_ip_address < forti_shutdown_cmds.txt

iii.) change unix permissions as needed for shutdownfg.sh and run it with 

./shutdownfg.sh

LAST UPDATE/EDIT:

Api key or SSH key the degree of risk appears to be the same. Either way an admin user with a minimum a custom permission is powerful. Can lock down limit its use to a dedicated interface on the FortiGate and other restrictions:

This requires leaving an admin's private ssh key for the FortiGate sitting on a largely unattended but isolated monitoring device (in my case at least) with the key in non-password-protected form, and this may not be as secure as the use of the api key.

OTHER SOLUTIONS:

Execute the custom switch commands and the FortiGate shutdown command via SSH shell script (instead Automation Stitch above):

Do the same as above but forti_shutdown_cmds.txt has contents:

execute switch-controller custom-command shutdown-sw-my <FortiSwitchF124SerialNumber>
execute switch-controller custom-command shutdown-sw-my <FortiSwitchF108ESerialNumber>
execute shutdown
y
exit

I get errors for both switches when running this, but they both shut down:

./shutdownfg.sh
FG # Command fail. Return code -13
FG # Command fail. Return code -13
FG # Connection to >FG_ip_address> closed by remote host.

Likewise, the 3 commands above can be defined in a CLI Automation Action triggered by an Incoming Webhook.

@Markus_M  provided the info requested in the original post in that there are not other commands/syntax to properly shut down FortiSwitches. Also the preference for the SSH shell script.

If anything that needs to be corrected here, please comment. 

Also would like suggestion from anyone on whether this post or Markus' is the solution overall (I don't really care, just whatever is most helpful to future users looking for such info)

Thanks

[Markus_M]

Wow, this is an amazing post and content. I will try to get more info about the webhook, whereas I think you mean restAPI.

Regarding the SSH key changes - these should not change on the FGT, the SSH program would notice that and alert me, but it has not happened since now maybe two years. I did not get any complaint from my SSH devices.

Then the private key on some isolated device. That is correct, however, you can leave a password on the private key. It would be asked at the start of the script you might be running, making it semi-interactive and more secure.

I believe the webhooks can be also rather powerful, although with the CLI you can do basically everything, the restAPI will be limited in its possible action, usable by attackers.

One thing that could be interesting to you, so I shall mention it:

"config system auto-script"

This can store some FortiGate Syntax scripts. They were in the distant past limited on characters, they may not be anymore.

Read more about it here:

https://community.fortinet.com/t5/FortiGate/Technical-Tip-Automated-script-execution/ta-p/193685

The scripts can be multi-line, like config vdom, edit root, config firewall .... Best is to prepare that stuff in a notepad and then copy and paste to FGT CLI.

Example - the bold part is multi-line:

Spoiler

config system auto-script
    edit "session-list"
        set interval 5
            set repeat 0
            set script "diag sys session filter src 10.10.10.48
            diag sys session list"
                set output-size 100
            next
        end

end

You could combine this even with the "alias" command. Alias "something" runs whatever is stored in "something". This can be the execuction for your auto-script (exec auto-script start session-list) in this example.

One thing to note about the auto-script, for anyone who is reading it. The "output-size" controls pretty much how much of the script output is recorded. It is recorded in RAM. Do not run this with recording 1GB of output for whatever purpose while not having enough RAM on the FortiGate - This could lead to a memory conserve mode.

Best regards,

Markus

[Damien1]

>>info about the webhook, whereas I think you mean restAPI.

Yes, I believe it ultimately comes down to how much the admin associated with the api key can be restricted while allowing shutdown. I believe I  misremembered reading something about being about to restrict the API to a more granular degree than may be the case. But yes, otherwise it seems in an unrestricted form it can grant as much as, or nearly as much access as ssh.  

Thanks for the added info here. 

UPDATE:

For either SSH or API Admin Key needed for an Incoming Webhook automation trigger, it looks like the admin profile with the minimual granted Access Control permissions in order to shut down and/or reboot a FortiGate is to create an Admin Profile :

--set System:Custom, and then Configuration:'Read/Write'  

--set all other Access Control permissions to 'None'

or CLI:

config system accprofile
    edit "shutdown_from_ups_admin"
        set comments ""
        set secfabgrp none
        set ftviewgrp none
        set authgrp none
        set sysgrp custom
        set netgrp none
        set loggrp none
        set fwgrp none
        set vpngrp none
        set utmgrp none
        set wifi none
        set admintimeout-override disable
        set system-diagnostics enable
        config sysgrp-permission
            set admin none
            set upd none
            set cfg read-write
            set mnt none
        end
    next
end

Is this understanding correct, as far as the minimum permissions required?

Is there a way of further restricting without granting full write permissions to system config and allowing access to the set of exec commands as well?

(I'm assuming what's allowed in the CLI via SSH with this profile is also allowed using the REST API but I'm not sure)

Thanks!

[Markus_M]

Hi Damien,

sorry for the slow responses.

I believe this should be good already. You cannot narrow it down further; if you have access to reboot the whole box already, you are able to affect that network severely already.

One thing to add, if you wanted to restrict that admin from logging into the SSH maybe you could disallow SSH from that respective interface (generally a good idea to not enable everything anywhere).

Best regards,

Markus

[Damien1]

Thanks @Markus_M, my response here is a bit slow.

In terms of strictly answering the question of the post "Native and Remote shutdown of managed FortiSwitch? (without SSHing into it from FortiGate)", I suppose the automation stitch to auto shut down the FortiSwitches whenever the FortiGate is shut down does address that, but I'm also loathe to mark my own comment about what I later did some testing with as the answer/solution to the question lol.

It does have your your ssh script approach too though as a good option that can be used in conjunction with it, so it might be the logical answer.

More importantly, as there isn't really a singular answer here since the conversation covered a few different topics, I've got all the info I needed and learned a number of things in the process of our discussion here.

Thanks for taking the time and sharing your knowledge here.

Much appreciated!