Description
This article describes five key points to consider when performing firmware upgrades in FortiOS and in general.
This applies to almost any change performed in a production environment.
This article gives some firmware upgrade tips.
Scope
FortiGate.
Solution
Understanding the new version first.
Before attempting any changes in production, first make sure to set up a laboratory where it will be possible to freely play with the new features to understand them with enough time and no pressure.
Read the release notes, manuals, and other documentation like presentations, videos, or podcasts about the new version.
Explain the need for an upgrade after understanding:
- The differences and the enhancements between the new version and the previous version(s).
- The impact of the upgrade on customers and the users of the operating platform.
- The known limitations that might affect the environment.
- The potential risks when performing the upgrade.
- The licensing changes that may apply.
Have a valid reason to upgrade.
The reason should NOT simply be a desire to have the latest version. The reason has to be explained in terms of business, technical, and/or operational improvement.
Affirmative answers to the following considerations are valid reasons to upgrade:
Affirmative answers to the following considerations are valid reasons to upgrade:
- Whether or not the new version have a feature that helps to ensure compliance.
- Whether the new version has an enhancement that allows a 40% decrease (40% improvement) on the time to perform a certain operation.
- Whether the new version corrects a known defect/bug found on a previous version that affects the company business/operations.
- Whether the new version allows the organization to deploy new services that will help to gain new customers or increase loyalty of existing ones.
- Whether the vendor is cutting support for the version that the organization is currently using.
If the best reason to upgrade is 'Because the new features seem to be cool' or simply to have the latest version for novelty's sake, more understanding and planning may be necessary.
Prepare an upgrade plan.
Make sure to create a plan that covers business, technical, and operational aspects of the upgrade.
Business:
Proper planning and justification for an upgrade have to be proportional to how critical the system is to the business.
- Make sure to clearly articulate the benefits of the upgrade in business terms (time, money, and efficiency).
- Understand the business processes that will be affected by the change.
- Make sure the upgrade maintenance window is not close to a business-critical process (such as quarterly or monthly business closure).
- Obtain executive and operational approval for the maintenance window. The approval must come from the owners of ALL the systems/information affected by the upgrade, not only from those that own the system being upgraded. The approval must be done in a formal (written or e-mail) form.
Technical and operational:
- Re-read the Release Notes for the technology upgrade. Supported hardware models, upgrade paths, and known limitations should be clearly understood.
- Always follow the upgrade path as described in the upgrade path tool article: Technical Tip: Difference in the upgrade path
- Make sure to upgrade maintenance window does not overlap with any other maintenance window on the infrastructure.
- If any premium support is offered (such as TAM, or Premium Support), do a capacity planning exercise to ensure the new firmware/software version does not take more hardware resources than possessed.
- Create a backup, whether or not backups are scheduled. Create a new fresh backup.
- Obtain offline copies of both the currently installed firmware and the new version.
- Create a list of systems with inter-dependencies to the system upgraded. For example, if a FortiGate is upgraded; understand the impact on any FortiAP, FortiAuthenticator, FortiToken, FortiManager, or FortiAnalyzer environment.
- Make sure to have a list of adjacent units to the upgrading platform and have administrative access to them, just in case it is needed to do some troubleshooting. Make sure to administratively access the Web Applications. Make sure to administratively access the surrounding switches and routers.
- Have a step-by-step plan on how to perform and test the upgrade. Make sure to think of the worst situation before it happens, and have predefined courses of action, instead of thinking under pressure when something has already gone wrong.
- Define a set of tests (that include critical business applications that should be working) to make sure the upgrade proceeded as expected. If any test does not go well, define which ones mandate a rollback and which ones can be tolerated for further troubleshooting. This set of tests should be run before and after the upgrade to compare results, and they should be the same.
- Define a clear rollback plan. If something goes wrong with the upgrade or the tests, the rollback plan will help to get the environment back to a known and operational status. The plan must clearly state the conditions under which the rollback will be started.
- Declare configuration freezes a short amount of time before and after the upgrade. The idea is to reduce the amount of variables to take into consideration if something goes wrong.
- Perform a 'Quality Assurance' upgrade. Grab a copy of the production configuration, load it on a non-production box and execute the upgrade there to see if there are any issues with the process. After, adjust plans according to the results obtained.
- Have a list of information elements to be gathered if something goes wrong. This ensures that, even if the upgrade fails, it will be possible to collect enough information to troubleshoot the issue without needing to repeat the problem. Get help from Fortinet Support if it is necessary to check what else could be missing on the list.
- Define a test monitoring period after the change was completed. Even if the upgrade went smoothly, something could still go wrong. Make sure to monitor the upgraded system for at least one business cycle. Business cycles may be a week, a month, or a quarter, depending on the organization’s business priorities.
Execute the upgrade plan.
Execution of an upgrade is just as key as planning.
Once the upgrade is performed, the pressure will rise and stress might peak.
This is why it is necessary to stick to the plan created with a cool head.
Resist the temptation to make decisions while performing the upgrade, as the judgment will be clouded by the stress of the moment, even if a new decision seems to be 'obvious' at such a time.
If the plan says to roll back, then execute the rollback despite the potential 'We-can-fix-this-very-quickly' mentality.
While performing the upgrade, make sure all the involved components are permanently monitored before, during, and after the upgrade, either via monitoring systems, SNMP alerts, or at least with tools like a ping.
Critical resources like CPU, memory, network, and/or disk utilization must also be constantly monitored.
To avoid misunderstandings: while performing the tests for each critical application defined on the planning, make sure there are formal notifications on the results for each user area, service, system, and/or application tested.
Regardless of whether rollback is necessary: if a problem occurs, make sure to gather as much information about the problem as possible in order to submit a Support ticket to find a solution.
Regardless of whether rollback is necessary: if a problem occurs, make sure to gather as much information about the problem as possible in order to submit a Support ticket to find a solution.
Last but not least, document the upgrade:
- Enable the terminal emulation program to leave a trace of all the commands executed and all the output generated. If the steps are performed via the GUI, consider using a video capture tool to document it.
- Document any command or change performed over the adjacent/interdependent systems. Make sure they are acknowledged by the relevant administrators
- Document any deviations performed over the upgrade plan. This is planned-versus-actual.
Learn more about change management.
Change Management and Change Control are huge knowledge areas in the field of Information Systems and Computer/Network Security.
This document is by no means a comprehensive list on what to do when performing an upgrade, with either Fortinet or any other technology. It is only a list of important things to take into consideration when performing upgrades which are the result of years of experience dealing with changes in critical environments, as it is common that security units are protecting critical applications and processes.
There are numerous other resources on the topic: books, public white papers, blog entries, etc.
Search the Internet for the 'Change Control Best Practices' or 'Change Management Best Practices' to see other resources.
