Native and Remote shutdown of managed FortiSwitch? (without SSHing into it from FortiGate)

Damien1 · ‎02-24-2022

Is there any way to properly shutdown a FortiGate-manged FortiSwitch without having to ssh into it from the FortiGate CLI?

Also, how about doing so remotely with FortiCloud?

(There's a way to reboot FortiSwitches from the FortiGate GUI and CLI, but not easily from FortiGate Cloud and not at all from FortiExplorer iOS)

The best thing I have been able to come up with is not very convient for remote use and it only sort of works:

1.) On the FortiGate define a custom-command to send to each of my 2 FortiSwitches:

config switch-controller custom-command
    edit "shutdown-sw-custom"
        set description "shutdown now script"
        set command "execute shutdown"
    next
end

2.) Invoke the commands from the ForiGate CLI, or from FortiGate Cloud define and deploy a script containing the commands:

execute switch-controller custom-command shutdown-sw-custom <FortiSwitchF124SerialNumber>
execute switch-controller custom-command shutdown-sw-custom <FortiSwitchF108ESerialNumber>

If it fully worked perfectly/reliably this would be an awkward but still usable workaround for a lack of a scriptable direct command for automation purposes, and also to a 'Shutdown FortiSwitch' button in the FortiGate GUI, FortiGate Cloud, and in FortiExplorer for manually invoked purposes... but when running the defined c ommand with my FortiSwitch F124 as the target I always get

Command fail. Return code -7621

response which halts the execution of a script...even though on the bright side anecdotally (small sample size) the switch shuts down anyway after a few minutes and/or after the FortiGate is shut down.

The command seems to work reliably for shutting down the FortiSwitch 108E FPOE without getting any error messages though.

Is there any way to shut down a managed FortiSwitch without having to SSH into it from the FortiGate in that is turn logged into from a computer locally or enabling remote access ?

Surely I'm overlooking a more simple way of shutting down a FortiSwitch manually and in an automation process?

Thanks!

(

Motivation:

Being able to shutdown FortiSwitches before shutting down the FortiGate 60F via automated script and manually in the event of a power outage and a monitoring device detects that the UPS' battery is getting low and other situtations...

command script triggered by the ForiGate shutdown process commencing, webhook from the UPS monitoring device

Running FortiOS 7.0.3 on all components.

)

Damien1 · ‎03-23-2022

FINAL UPDATES/EDITS ADDED in places below with corrections from other parts of the converstaions in this thread:

Okay, I hope this helps someone else too, so I'll give details on what I've best been able to determine:

Overall, the motivation of the original post is to verify that there is not a native command to shut down a managed FortiSwitch that I was not finding a reference to.
Markus answers that question as likely no (thanks for verifying that).

The workaround in the post using the custom switch commands initially seemed to be unreliable due to the error messages I encounter in running them both in an interactive FortiGate CLI prompt, and via a SSH script. One of the FortiSwitches was also initially to slow to respond to the commands as well, but that got faster with time for some reason.

Through a fair amount of testing by invoking these custom commands in various contexts, it appears that all of the error messages the FortiGate reports in response can be safely ignored (no errors are reported in the FortiGate logs).

The initial automation script failure on the FortiGate has not repeated itself either.

If there are errors going on, the FortiGate appears to be able to process them nonetheless and scripts do not exit in reaction to the errors.

I no longer have the reliability concerns in using the originally described workarounds.

The following is the description of the objective and proposed solutions to what motivated the initial desire to find a native/supported way of shutting down a managed FortiSwitch without ssh'ing into it from a ForiGate:

OBJECTIVE:
The method to shutdown FortiGate-managed FortiSwitches is to log into the FortiGate via GUI or ssh, and then ssh into each FortiSwitch from the FortiGate CLI.
This requires entering a password and local FortiLink ip address corresponding to each switch (admin is prompted to set a password on the first ssh login a FortiSwitch from the FortiGate)
e.g. FortiGate CLI: # ssh admin@169.254.1.x

For a remote and manual shutdown of FortiSwitches, this seems to require enabling remote access on the FortiGate, as only restart buttons presently exist for FortiSwitches in the FortiGate GUI, FortiCloud GUI, and FortiExplorer.

The objective is to reduce the number of steps involved in shutting down FortiSwitches before shutting down the FortiGate in both manual and in automated manners, and to be able to do so remotely and locally.

An example use case is in the event of a power outage or approaching storm on an aging power grid known to have problems during weather events (U.S. power grid is aging). One may wish to be able quickly shut down a FortiGate and connected FortiSwitches at small business or in a home that are connected to a UPS of a limited battery capacity compared to a large UPS one might find in an enterprise Fortinet setup (assumes that the ISP cable modem, etc. is powered by the UPS as well).

One may wish to do so manually and/or in an automated manner when the monitoring device attached to the UPS detects the UPS battery to be in use or is at a low remaining capacity depending on the situation.

All solutions make use of creating the custom command that is applied to each FortiSwitch as shown in original post:

1.) On the FortiGate define a custom-command to send to each FortiSwitch:

config switch-controller custom-command
    edit "shutdown-sw-custom"
    set description "shutdown now script"
    set command "execute shutdown"
    next
end

2.) The commands can be executed on the FortiGate via an Automation Action CLI script with any available trigger (e.g. incoming webhook)
or in an shell script via SSH from an external device in a few different combinations described in the solutions below:

execute switch-controller custom-command shutdown-sw-custom <FortiSwitchF124SerialNumber>
execute switch-controller custom-command shutdown-sw-custom <FortiSwitchF108ESerialNumber>

The particular failure/error messages encountered in using them with my FortiGate and managed FortiSwitches are noise that should be ignored as far as I have been able to determine
(FortiGate 60F, FortiSwitch 108E-FPOE, FortiSwitch 124F, FortiOS 7.0.3 on all)

SOLUTION 1 - UNIVERSAL SOLUTION FOR ALL USE CASES: ANY DESIRED METHOD TO SHUT DOWN A FORTIGATE ALSO SHUTS DOWN ITS MANAGED FORTISWITCHES

Create an Automation Stitch that detects in the logs that the FortiGate has received a shutdown command, and have the FortiGate shutdown its FortiSwitches first.

--Enables one to be able to easily manually shut down the switches and managing FortiGate via FortiExplorer phone app, FortiCloud GUI, FortiGate GUI, ...
--Reduces an ssh shell script or ssh command to a single command (FortiGate shutdown command), as well as being of use with other Automation Stitches
--Is the solution that shuts down the FS's and FG the most quickly as the shutdown commands for each switch are executed in parallel
--The most simple to use

FortiGate CLI:

config system automation-action
    edit "ShutdownSwitchFS1Action"
        set description "runs switch shutdown custom-command target FS 108E"
        set action-type cli-script
        set minimum-interval 0
        set script "execute switch-controller custom-command shutdown-sw-custom <FortiSwitchF108ESerialNumber>"
        set execute-security-fabric disable
        set accprofile "super_admin"
    next
    edit "ShutdownSwitchFS2Action"
        set description "runs switch custom-command target FS 124F"
        set action-type cli-script
        set minimum-interval 0
        set script "execute switch-controller custom-command shutdown-sw-custom <FortiSwitchF124SerialNumber>"
        set execute-security-fabric disable
        set accprofile "super_admin"
    next
end
config system automation-trigger
    edit "OnFortiGateShutdown"
        set description ''
        set trigger-type event-based
        set event-type event-log
        set logid 32200
    next
end
config sys automation-stitch
    edit "ShutdownSwitchesOnFgShutdown"
        set description ''
        set status enable
        set trigger "OnFortiGateShutdown"
        config actions
            edit 1
                set action "ShutdownSwitchFS1Action"
                set delay 0
                set required disable
            next
            edit 2
                set action "ShutdownSwitchFS2Action"
                set delay 0
                set required disable
            next
        end
    next
end

Notes:
--Can use a more restricted admin account type with minimum privileges required for added security LAST UPDATE/EDIT: (yes, to some extent, but there will still be a lot of access granted by it)

--Can combine both actions into one. I chose to use 2 separate actions in case one were to fail and also take advantage of speed by their execution in parallel processes

USING SOLUTION 1 WITH A DEVICE MONITORING A UPS THAT IS CONNECTED TO A ETHERNET PORT ON THE FORTIGATE

A.) 'Incoming Webhook' Automation Trigger to trigger the FortiGate shutdown:
~~Possibly the most secure method?~~ If the FortiGate is set up with a public or private CA (i.e. such that no browser warnings about the SSL/TLS connection to the FortiGate GUI is insecure), then the connected client device and FortiGate can verify one another's identity with the security certificates.

LAST UPDATE/EDIT: Thanks @Markus_M for clarifying that this does not pose any issues so crossing this out (got the initial misimpression that the SSH key can periodically change from an older post that no longer applies)
If I understand correctly, the SSH host key on the FortiGate changes periodically, so a shell script would have to be set for the ssh connection to Not verify the identity of the host to guarantee it will not be interrupted with a warning prompt.

OR

B.) Run execute shutdown command on the FortiGate via SSH. Numerous ways to do this, but in line with the method that Markus shows, on the monitoring device.

i) create file forti_shutdown_cmds.txt with the contents:

execute shutdown
y
exit

(probably best to make sure that the new line characters are new line \n only and not Windows \r\n)

ii.) create shell script shutdownfg.sh with contents:

ssh -T -o "StrictHostKeyChecking=no" -p <FG_SSH_PORT> AdminName@FG_ip_address < forti_shutdown_cmds.txt

iii.) change unix permissions as needed for shutdownfg.sh and run it with

./shutdownfg.sh

LAST UPDATE/EDIT:

Api key or SSH key the degree of risk appears to be the same. Either way an admin user with a minimum a custom permission is powerful. Can lock down limit its use to a dedicated interface on the FortiGate and other restrictions:

This requires leaving an admin's private ssh key for the FortiGate sitting on a largely unattended but isolated monitoring device (in my case at least) with the key in non-password-protected form, ~~and this may not be as secure as the use of the api key.~~

OTHER SOLUTIONS:

Execute the custom switch commands and the FortiGate shutdown command via SSH shell script (instead Automation Stitch above):

Do the same as above but forti_shutdown_cmds.txt has contents:

execute switch-controller custom-command shutdown-sw-my <FortiSwitchF124SerialNumber>
execute switch-controller custom-command shutdown-sw-my <FortiSwitchF108ESerialNumber>
execute shutdown
y
exit

I get errors for both switches when running this, but they both shut down:

./shutdownfg.sh
FG # Command fail. Return code -13
FG # Command fail. Return code -13
FG # Connection to >FG_ip_address> closed by remote host.

Likewise, the 3 commands above can be defined in a CLI Automation Action triggered by an Incoming Webhook.

@Markus_M provided the info requested in the original post in that there are not other commands/syntax to properly shut down FortiSwitches. Also the preference for the SSH shell script.

If anything that needs to be corrected here, please comment.

Also would like suggestion from anyone on whether this post or Markus' is the solution overall (I don't really care, just whatever is most helpful to future users looking for such info)

Thanks

View solution in original post

EEHC · ‎04-13-2022

I want to say the same words "Wow, this is an amazing post and content."

EEHC

Damien1 · ‎04-27-2022

Thanks @EEHC the more we all share in creating tribal knowledge in addition to the recommended best practices, the better off we all are :)

Anthony_E · ‎03-14-2022

Hello Damien,

Sorry for the confusion.

I will find you somebody with the skill to answer your question as soon as possible.

Thanks a lot for your patience.

Regards,

Anthony-Fortinet Community Team.

Anthony_E · ‎03-18-2022

Hello Damien,

I just come back to you to indicate you that we are still investigating and are trying to find an answer to your question.

Thanks a lot for your patience.

Regards,

Anthony-Fortinet Community Team.

Damien1 · ‎03-18-2022

No problem, I understand and thank you.

I formed the notion that a FortiSwitch needs to be shut down after I happened to come across the warning to do so in the FortiSwitchOS CLI Reference doc next to the 'execute shutdown' command syntax.

(Necessity to shut down is not mentioned in the Managed Switch doc or the Standalone Switch doc, so I assumed since no distinction was made that this warning applies to both managed and standalone FortiSwitches.)

So just to make sure out of respect for the time and resources that may be involved on FortiNet's end, I would like to verify my impressions that:

1.) A FortiSwitch managed by FortiGate needs to be cleanly shut down before loss of power or disconnection from power

If so, then:
2.) This is only due to the integrity of the switch config file, and power loss does
Not risk damaging the switch's hardware? (did not find a risk to hardware in docs)

If so (and no hardware health risk):
3.) What's needed to restore functionality to the switch and its config?
Is the config restored by the FortiGate managing it automatically (or perhaps the FortiGate pushes the config automatically every time it boots or something already), or is manual intervention required?

If there is only a risk of config file loss as a power loss consequence to consider, that risk is known to be very low, and the config is auto fixed by the managing FortiGate anyway, this would be helpful to inform how important the shut down process is relative to a

particular use case.

Thank you

Markus_M · ‎03-19-2022

Hello Damien,

once the FortiSwitch and FortiGate are paired, they will continue to communicate via FortiLink. Shutting down the switch is cautious, but I have not seen it complaining about it being power cycled. No hardware risk I can think of due to abrupt shutdown.

From another perspective: It should be able to resist problems like that, for the single reason that there are actually PD switches, they get power delivery over PoE, 108E for example. If the power is lost from another PoE switch maybe, it must be able to come back up and restore its information. That is the case in my lab.

FortiGate > 108D PoE > 108E. If the 108D reboots, the 108E will be power cycled.

If on GUI I select to reset PoE, the 108E will be power cycled.

Besides, config wise, it won't do much that would be in risk of loss for a power loss. The switch config is rather static, the dynamic VLANs, if any, will be written by FortiGate in runtime. Active RADIUS authentication, NAC will decide on that during runtime.

Hardware wise, any device (not only Fortinet) can fail due to age of components.

Example:

You shut it down, it does not start anymore because the conditions of start are harder than conditions of runtime. This, however, would be no different from shutdown vs power plug.

One thing that the later FortiOS are more resilient to are file system errors.

When you power cycle a file system, it might not like this and will want to check the integrity due to incomplete write operations during the power loss. This will be done automatically.

And kudo to you for posting so detailed, that is rare and a joy to read. Many do not know what their question actually is, but ask anyway. Then one has to start with guesswork.

Best regards,

Markus

Damien1 · ‎03-23-2022

Thanks for all of this info, this is really interesting in learning about all of this.

>>You shut it down, it does not start anymore because the conditions of start are harder than conditions of runtime. This, however, would be no different from shutdown vs power plug.

Ah, this reminds me of what a computer engineer friend of mine explained to me once about about HDD spinning hard drives: he explained to not make the number of minutes before spin-down too short, because while spinning down can preserve longevity, when they spin up the in-rush of electrical energy through all of the components in the HDD causes thermal expansion which puts stresses wear and tear on it as well, so it is best to seek a balance when attempting to optimize their use for long-term reliability in a backup system, etc.

>> And kudo to you for posting so detailed, that is rare and a joy to read.

Thank you that is kind. I was a bit worried that the posts might have too much detail. I'm here to learn.

Native and Remote shutdown of managed FortiSwitch? (without SSHing into it from FortiGate)

Nominate a Forum Post for Knowledge Article Creation