Skip to main content
Damien1
Explorer
February 25, 2022
Solved

Native and Remote shutdown of managed FortiSwitch? (without SSHing into it from FortiGate)

  • February 25, 2022
  • 16 replies
  • 27226 views

Is there any way to properly shutdown a FortiGate-manged FortiSwitch without having to ssh into it from the FortiGate CLI? 

Also, how about doing so remotely with FortiCloud?

 

(There's a way to reboot FortiSwitches from the FortiGate GUI and CLI, but not easily from FortiGate Cloud and not at all from FortiExplorer iOS)

 

The best thing I have been able to come up with is not very convient for remote use and it only sort of works:

 

1.) On the FortiGate define a custom-command to send to each of my 2 FortiSwitches:

 

 

config switch-controller custom-command     edit "shutdown-sw-custom"         set description "shutdown now script"         set command "execute shutdown"     next end

 

 

 

2.) Invoke the commands from the ForiGate CLI, or from FortiGate Cloud define and deploy a script containing the commands:

 

 

execute switch-controller custom-command shutdown-sw-custom <FortiSwitchF124SerialNumber> execute switch-controller custom-command shutdown-sw-custom <FortiSwitchF108ESerialNumber>

 

 

 

If it fully worked perfectly/reliably this would be an awkward but still usable workaround for a lack of a scriptable direct command for automation purposes, and also to a 'Shutdown FortiSwitch' button in the FortiGate GUI, FortiGate Cloud, and in FortiExplorer for manually invoked purposes... but when running the defined c ommand with my FortiSwitch F124 as the target I always get

Command fail. Return code -7621

response which halts the execution of a script...even though on the bright side anecdotally (small sample size) the switch shuts down anyway after a few minutes and/or after the FortiGate is shut down.

 

The command seems to work reliably for shutting down the FortiSwitch 108E FPOE without getting any error messages though.

 

Is there any way to shut down a managed FortiSwitch without having to SSH into it from the FortiGate in that is turn logged into from a computer locally or enabling remote access ?

 

Surely I'm overlooking a more simple way of shutting down a FortiSwitch manually and in an automation process?

 

Thanks!

 

(

 

Motivation:

Being able to shutdown FortiSwitches before shutting down the FortiGate 60F via automated script and manually in the event of a power outage and a monitoring device detects that the UPS' battery is getting low and other situtations...

command script triggered by the ForiGate shutdown process commencing, webhook from the UPS monitoring device

 

Running FortiOS 7.0.3 on all components.

)

 

 

 

Best answer by Damien1

FINAL UPDATES/EDITS ADDED in places below with corrections from other parts of the converstaions in this thread:

 

Okay, I hope this helps someone else too, so I'll give details on what I've best been able to determine: 

 

Overall, the motivation of the original post is to verify that there is not a native command to shut down a managed FortiSwitch that I was not finding a reference to.
Markus answers that question as likely no (thanks for verifying that).

The workaround in the post using the custom switch commands initially seemed to be unreliable due to the error messages I encounter in running them both in an interactive FortiGate CLI prompt, and via a SSH script. One of the FortiSwitches was also initially to slow to respond to the commands as well, but that got faster with time for some reason.

 

Through a fair amount of testing by invoking these custom commands in various contexts, it appears that all of the error messages the FortiGate reports in response can be safely ignored (no errors are reported in the FortiGate logs).

The initial automation script failure on the FortiGate has not repeated itself either.


If there are errors going on, the FortiGate appears to be able to process them nonetheless and scripts do not exit in reaction to the errors.

I no longer have the reliability concerns in using the originally described workarounds.

 

The following is the description of the objective and proposed solutions to what motivated the initial desire to find a native/supported way of shutting down a managed FortiSwitch without ssh'ing into it from a ForiGate:

 

OBJECTIVE:
The method to shutdown FortiGate-managed FortiSwitches is to log into the FortiGate via GUI or ssh, and then ssh into each FortiSwitch from the FortiGate CLI.
This requires entering a password and local FortiLink ip address corresponding to each switch (admin is prompted to set a password on the first ssh login a FortiSwitch from the FortiGate)
e.g. FortiGate CLI: # ssh admin@169.254.1.x

 

For a remote and manual shutdown of FortiSwitches, this seems to require enabling remote access on the FortiGate, as only restart buttons presently exist for FortiSwitches in the FortiGate GUI, FortiCloud GUI, and FortiExplorer.

 

The objective is to reduce the number of steps involved in shutting down FortiSwitches before shutting down the FortiGate in both manual and in automated manners, and to be able to do so remotely and locally.

 

An example use case is in the event of a power outage or approaching storm on an aging power grid known to have problems during weather events (U.S. power grid is aging). One may wish to be able quickly shut down a FortiGate and connected FortiSwitches at small business or in a home that are connected to a UPS of a limited battery capacity compared to a large UPS one might find in an enterprise Fortinet setup (assumes that the ISP cable modem, etc. is powered by the UPS as well).


One may wish to do so manually and/or in an automated manner when the monitoring device attached to the UPS detects the UPS battery to be in use or is at a low remaining capacity depending on the situation.


All solutions make use of creating the custom command that is applied to each FortiSwitch as shown in original post:

 

1.) On the FortiGate define a custom-command to send to each FortiSwitch:

 

 

config switch-controller custom-command     edit "shutdown-sw-custom"     set description "shutdown now script"     set command "execute shutdown"     next end

 

 


2.) The commands can be executed on the FortiGate via an Automation Action CLI script with any available trigger (e.g. incoming webhook)
or in an shell script via SSH from an external device in a few different combinations described in the solutions below:

 

 

execute switch-controller custom-command shutdown-sw-custom <FortiSwitchF124SerialNumber> execute switch-controller custom-command shutdown-sw-custom <FortiSwitchF108ESerialNumber>

 

 


The particular failure/error messages encountered in using them with my FortiGate and managed FortiSwitches are noise that should be ignored as far as I have been able to determine
(FortiGate 60F, FortiSwitch 108E-FPOE, FortiSwitch 124F, FortiOS 7.0.3 on all)

 

 

SOLUTION 1 - UNIVERSAL SOLUTION FOR ALL USE CASES: ANY DESIRED METHOD TO SHUT DOWN A FORTIGATE ALSO SHUTS DOWN ITS MANAGED FORTISWITCHES

Create an Automation Stitch that detects in the logs that the FortiGate has received a shutdown command, and have the FortiGate shutdown its FortiSwitches first.

--Enables one to be able to easily manually shut down the switches and managing FortiGate via FortiExplorer phone app, FortiCloud GUI, FortiGate GUI, ...
--Reduces an ssh shell script or ssh command to a single command (FortiGate shutdown command), as well as being of use with other Automation Stitches
--Is the solution that shuts down the FS's and FG the most quickly as the shutdown commands for each switch are executed in parallel
--The most simple to use

 

FortiGate CLI:

 

 

config system automation-action     edit "ShutdownSwitchFS1Action"         set description "runs switch shutdown custom-command target FS 108E"         set action-type cli-script         set minimum-interval 0         set script "execute switch-controller custom-command shutdown-sw-custom <FortiSwitchF108ESerialNumber>"         set execute-security-fabric disable         set accprofile "super_admin"     next     edit "ShutdownSwitchFS2Action"         set description "runs switch custom-command target FS 124F"         set action-type cli-script         set minimum-interval 0         set script "execute switch-controller custom-command shutdown-sw-custom <FortiSwitchF124SerialNumber>"         set execute-security-fabric disable         set accprofile "super_admin"     next end config system automation-trigger     edit "OnFortiGateShutdown"         set description ''         set trigger-type event-based         set event-type event-log         set logid 32200     next end config sys automation-stitch     edit "ShutdownSwitchesOnFgShutdown"         set description ''         set status enable         set trigger "OnFortiGateShutdown"         config actions             edit 1                 set action "ShutdownSwitchFS1Action"                 set delay 0                 set required disable             next             edit 2                 set action "ShutdownSwitchFS2Action"                 set delay 0                 set required disable             next         end     next end

 

 


Notes:
--Can use a more restricted admin account type with minimum privileges required for added security LAST UPDATE/EDIT: (yes, to some extent, but there will still be a lot of access granted by it)

--Can combine both actions into one. I chose to use 2 separate actions in case one were to fail and also take advantage of speed by their execution in  parallel processes

 

 

USING SOLUTION 1 WITH A DEVICE MONITORING A UPS THAT IS CONNECTED TO A ETHERNET PORT ON THE FORTIGATE

A.) 'Incoming Webhook' Automation Trigger to trigger the FortiGate shutdown:
Possibly the most secure method? If the FortiGate is set up with a public or private CA (i.e. such that no browser warnings about the SSL/TLS connection to the FortiGate GUI is insecure), then the connected client device and FortiGate can verify one another's identity with the security certificates.

 

LAST UPDATE/EDIT: Thanks @Markus_M for clarifying that this does not pose any issues so crossing this out (got the initial misimpression that the SSH key can periodically change from an older post that no longer applies)
If I understand correctly, the SSH host key on the FortiGate changes periodically, so a shell script would have to be set for the ssh connection to Not verify the identity of the host to guarantee it will not be interrupted with a warning prompt.

 

OR


B.) Run execute shutdown command on the FortiGate via SSH. Numerous ways to do this, but in line with the method that Markus shows, on the monitoring device.

 

i) create file forti_shutdown_cmds.txt with the contents:

 

 

execute shutdown y exit

 

 

(probably best to make sure that the new line characters are new line \n only and not Windows \r\n)

 

ii.) create shell script shutdownfg.sh with contents:

 

 

ssh -T -o "StrictHostKeyChecking=no" -p <FG_SSH_PORT> AdminName@FG_ip_address < forti_shutdown_cmds.txt

 

 

 

iii.) change unix permissions as needed for shutdownfg.sh and run it with 

 

 

./shutdownfg.sh

 

 

LAST UPDATE/EDIT:

Api key or SSH key the degree of risk appears to be the same. Either way an admin user with a minimum a custom permission is powerful. Can lock down limit its use to a dedicated interface on the FortiGate and other restrictions:

This requires leaving an admin's private ssh key for the FortiGate sitting on a largely unattended but isolated monitoring device (in my case at least) with the key in non-password-protected form, and this may not be as secure as the use of the api key.

 


OTHER SOLUTIONS:

Execute the custom switch commands and the FortiGate shutdown command via SSH shell script (instead Automation Stitch above):

Do the same as above but forti_shutdown_cmds.txt has contents:

 

 

execute switch-controller custom-command shutdown-sw-my <FortiSwitchF124SerialNumber> execute switch-controller custom-command shutdown-sw-my <FortiSwitchF108ESerialNumber> execute shutdown y exit

 

 

 

I get errors for both switches when running this, but they both shut down:

 

 

./shutdownfg.sh FG # Command fail. Return code -13 FG # Command fail. Return code -13 FG # Connection to >FG_ip_address> closed by remote host.

 

 

 

 

Likewise, the 3 commands above can be defined in a CLI Automation Action triggered by an Incoming Webhook.

 

@Markus_M  provided the info requested in the original post in that there are not other commands/syntax to properly shut down FortiSwitches. Also the preference for the SSH shell script.

 

If anything that needs to be corrected here, please comment. 

Also would like suggestion from anyone on whether this post or Markus' is the solution overall (I don't really care, just whatever is most helpful to future users looking for such info)

 

Thanks

16 replies

Anthony_E
Staff
Staff
February 27, 2022

Hello Damien,

 

May I propose you to have a look in our Knowledge Base?

You can make research via the search bar, selecting Knowledge base.

 

https://community.fortinet.com/t5/Knowledge-Base/ct-p/knowledgebase

 

If you do not find the solution there, somebody will help you.

 

Regards,

Best Regards
Damien1
Damien1Author
Explorer
March 10, 2022

>>May I propose you to have a look in our Knowledge Base?

>>You can make research via the search bar, selecting Knowledge base.

 

No doubt people often want to skip reading the documentation and spending the time exploring existing resources (and thereby learning in general in the process) that people have put a lot of work into creating. Prior to this posting I did rather carefully go through literally all of the available documentation for FortiGate units and FortiSwitch units, as well as FortiCloud-related, and I spent time in the Knowledge Base over a period of a few days in addition to the forum. Also checked the FortiNet sub-Reddit, FortiNet YouTube channel. Googled in general.

 

In doing so, I found some bits of info in both the Knowledge Base and in the Forum of what others have shared that are partially related to some aspects of what I mentioned that I was trying to accomplish and in what manner. But I did not find enough information to the extent that fully addressed what I hoped to accomplish overall, which motivated the time I put into composing my first post here and provide as much info as possible about what I have tried. 

(Also did not find anything at all in the resources that happened to matched the problems I ran into with the same error code response.)

 

That being said, it is entirely possible that I overlooked some information somewhere. Asking here after having exhausted all available resources, basically. 

If I missed something, I definitely would like to know.

Thanks.

Markus_M
Staff & Editor
Staff & Editor
March 18, 2022

Hello Damien,

 

i cannot think of a way of how to do it better without SSH.

But you can remotely execute SSH commands without logging in with username-password on the traditional way. From what I see from you, you might have worked out SSH key connection. When you can log in like that from an SSH key capable node (the UPS is such node maybe), then you can run the commands as such.

 

cat scripts/fortizoo.sh  
#!/bin/sh
#vars setup
#local:
FGTVAR="/root/scripts/fortigate.var"
#remote:
FSSH="ssh admin@192.168.200.1"

#create cert-import on FGT via FGTVAR
echo "config root" > $FGTVAR
echo "edit root" >> $FGTVAR

echo "execute switch-controller custom-command shutdown-sw-custom <FortiSwitchF124SerialNumber>" >> $FGTVAR
echo "exit" >> $FGTVAR
sleep 1
$FSSH < $FGTVAR
sleep 5
rm $FGTVAR
sleep 1

I use this script to propagate linux "dehydrated" letsencrypt certificates to a FortiMail and a FortiGate and it writes certs to both devices. I tried to adapt it for your case. It would run for a Linux machine as is, Windows mileage may vary.

On Windows you can run commands like this remotely via putty link (=plink.exe), but I lack experience with it (we do have a community article here with plink).

 

Best regards,

 

Markus

Damien1
Damien1Author
Explorer
March 18, 2022

UPDATE 2: next reply below with tested ways of achieving objective

 

UPDATE: in process of finishing more testing that's looking promising in either script-based approach...the fail errors only seem to happen in a interactive mode (CLI prompt and FG Cloud 'deploy script' and the complaining switch is shutting down more quickly. Can not reproduce the previous single automation stitch failure. SSH and CLI scripts seem to not exit with errors as expected. Will update shortly

 

 

 

Thanks for the input here and shell script example too.

--For shutting down locally in an automated manner:

Okay so if I understand correctly, it looks like the 2 options to shut down managed FortiSwitches are to either use custom switch commands or ssh into the switches by first ssh'ing into the FortiGate (or perhaps directly to the switches with some additional set up) 

I've been considering the sort of approach you outline with either a SSH script or 

an automation stitch action to run them in a CLI script on the FortiGate that is triggered on via webhook (LAN-only as well).

 

As described in the post, the stumbling block that I run into that I think applies to either approach:

Executing the custom command results in an error message code every time when the FortiSwitch F124 is the target (100% of the time when executed in the CLI console on the FortiGate GUI, although the switch shuts down after a while or after the FortiGate is shut down after that. The other switch shuts down immediately without a error code). 

 

To isolate this problem, for testing I also created 2 temporary time-based automation stitches to execute the custom command in a script for each of the 2 FortiSwitches. I also executed the custom commands by deploying scripts from FortiGate Cloud for a manual shutdown use case.

In either case these separately tested automations fail to shut down either switch  regardless of whether it was deployed from FortiGate Cloud or in the local time-based automation stitch.

UPDATE:  ^^^ Experienced this once and have not been able to reproduce this failure since.

 

So it seems that returned errors declaring failure to execute the custom switch command when manually invoked would also cause any CLI script using them (with required admin privileges) to be interrupted and fail to run to completion. It seems that a shell script sshing into the FortiGate and invoking the same custom switch command to shutdown the 124F will be interrupted due to this error, and it appears that it will fail on the 108E as well (I will test it to verify though)

 

Error codes examples: Command fail. Return code -13 or -7621 and others I've seen more since the original post here.

FortiGate Cloud returns error codes for either switch in a deployed script, and the CLI console on the Fortigate GUI returns an error code only for the 124F.

 

(The SSH approach also seems to require SSHing into the FortiGate while disabling the 'verify server authenticity' as the key on the FortiGate seems to change periodically with either RSA or ssh-ed25519-based keys from what I've read here and elsewhere perhaps after the FortiGate is updated but I'm not sure.)  

 

 

If it is more appropriate for me to file a support ticket please let me know and I can mark your solution as the solution. (I'm new to FortiNet)

 

 

--For shutting down remotely & manually:

Perhaps adding a shutdown button next to the existing reboot button for a FortiSwitch in FortiExplorer phone app and FortiGate Cloud may be a feature request for consideration if thought to be useful? The existing reboot and the shutdown and reboot buttons for the FortiGate is handy. 

(apologies if there's a feature request section on the website somewhere that I overlooked)

^^^ UPDATE: the automation stitch in the reply to this post takes care of this.

 

(In my case, I have used a Raspberry Pi Compute 3 up until acquiring the FortiNet components, on a dedicated VLAN with no internet access to monitor the UPS and auto shut down other computers via SSH, and a few other things via MQTT via a bash script when the UPS battery runs low...

I'm not a security expert, but my preference to use a webhook triggering a CLI script on the FortiGate approach is because I've read in the NSA leaks from a few years ago that private SSH keys that are not manually password protected are particularly high value targets in breaches, so an unencrypted private ssh key with admin privileges for the network firewall that is sitting on a device that will occasionally need to be connected to the internet to get updates does not seem ideal to me, but I could be overly concerned or mistaken here. My logic to use a webhook is that if its API key were to compromised, the most that an attacker could do with it is shut everything down. I will probably do the same on other connected computers that I can set up to react to local webhooks)

 

Thank you again.

 

Anthony_E
Staff
Staff
March 14, 2022

Hello Damien,

 

Sorry for the confusion.

 

I will find you somebody with the skill to answer your question as soon as possible.

 

Thanks a lot for your patience.

 

Regards,

Best Regards
Anthony_E
Staff
Staff
March 18, 2022

Hello Damien,

 

I just come back to you to indicate you that we are still investigating and are trying to find an answer to your question.

 

Thanks a lot for your patience.

 

Regards,

Best Regards
Damien1
Damien1Author
Explorer
March 18, 2022

No problem, I understand and thank you.

I formed the notion that a FortiSwitch needs to be shut down after I happened to come across the warning to do so in the FortiSwitchOS CLI Reference doc next to the 'execute shutdown' command syntax.

(Necessity to shut down is not mentioned in the Managed Switch doc or the Standalone Switch doc, so I assumed since no distinction was made that this warning applies to both managed and standalone FortiSwitches.)

 

So just to make sure out of respect for the time and resources that may be involved on FortiNet's end, I would like to verify my impressions that:

 

1.) A FortiSwitch managed by FortiGate needs to be cleanly shut down before loss of power or disconnection from power

 

If so, then:
2.) This is only due to the integrity of the switch config file, and power loss does
Not risk damaging the switch's hardware? (did not find a risk to hardware in docs)

 

If so (and no hardware health risk):
3.) What's needed to restore functionality to the switch and its config?
Is the config restored by the FortiGate managing it automatically (or perhaps the FortiGate pushes the config automatically every time it boots or something already), or is manual intervention required?


If there is only a risk of config file loss as a power loss consequence to consider, that risk is known to be very low, and the config is auto fixed by the managing FortiGate anyway, this would be helpful to inform how important the shut down process is relative to a

particular use case.

 

Thank you

Markus_M
Staff & Editor
Staff & Editor
March 19, 2022

Hello Damien,

 

once the FortiSwitch and FortiGate are paired, they will continue to communicate via FortiLink. Shutting down the switch is cautious, but I have not seen it complaining about it being power cycled. No hardware risk I can think of due to abrupt shutdown.

 

From another perspective: It should be able to resist problems like that, for the single reason that there are actually PD switches, they get power delivery over PoE, 108E for example. If the power is lost from another PoE switch maybe, it must be able to come back up and restore its information. That is the case in my lab.

FortiGate > 108D PoE > 108E. If the 108D reboots, the 108E will be power cycled.

If on GUI I select to reset PoE, the 108E will be power cycled.

 

Besides, config wise, it won't do much that would be in risk of loss for a power loss. The switch config is rather static, the dynamic VLANs, if any, will be written by FortiGate in runtime. Active RADIUS authentication, NAC will decide on that during runtime.

 

Hardware wise, any device (not only Fortinet) can fail due to age of components.

Example:

You shut it down, it does not start anymore because the conditions of start are harder than conditions of runtime. This, however, would be no different from shutdown vs power plug.

 

One thing that the later FortiOS are more resilient to are file system errors.

When you power cycle a file system, it might not like this and will want to check the integrity due to incomplete write operations during the power loss. This will be done automatically.

 

And kudo to you for posting so detailed, that is rare and a joy to read. Many do not know what their question actually is, but ask anyway. Then one has to start with guesswork.

 

Best regards,

 

Markus

Thought Leadership Security Summit. Outpace New Threats with AI - enhanced defense. Tuesday, Septmeber 15, 8:30 AM - 2:30 PM PT. The Golf Club at Newcastle, WA.
Fortinet Flag the Hack. Wednesday, August 26, 9:00 AM - 5:00 PM ET, COSM, Atlanta, GA.
Virtual event | September 2026. SASE summit. The age of autonomous trust. Register here!