FortiGate Scalable HA Architecture as Defined in Azure Marketplace

[MartTwom]

Components:

Azure Load Balancer – Abstracted Azure resource which is scalable and resilient. Dynamically splits traffic between the two FortiGates.

Virtual Network – 10.40.0.0/16, also known as VNET

Public Facing Network – 10.40.1.0/24

Protected Network – 10.40.2.0/24

Availability Set – Method of grouping resources within Azure to ensure that they are hosted on separate physical hardware in order to ensure that at any given time (even during upgrades and maintenance) at least one of the set will remain up.

FortiGate – Azure certified virtual appliance running the same OS which is used on our hardware appliances. These will be referenced as FortiGate-A and FortiGate-B.

FortiManager – Dedicated policy and configuration manager, used to keep the configuration in sync between the two FortiGates.

How to deploy:

Utilize the FortiGate HA template which is available in the Microsoft Azure Marketplace to deploy the network resources and FortiGates as depicted in the diagram.

Configuration:

Azure Load Balancer

All traffic coming from outside of Azure will pass through the load balancer first. The load balancer uses Network Address Translation and Port Address Translation (NAT/PAT) to connect a single public IP address to the Azure VNET. Within the Azure portal there are two options for configuring these NAT rules. The first is called “Inbound NAT rules.” The second is termed “Load balancing Rules”

Inbound NAT rules

These rules are applied to a specific host and are not load balanced. As such, these are typically used for management. The template uses ports 443 and 22 for management of FortiGate-A. Ports 8443 and 8022 are similarly directed at FortiGate-B. Once the FortiGates are configured, you can change these ports. For example, if you want to use port 443 for internal web services, you could configure an alternate port on FortiGate-A for management, and modify this rule to use that new port. Once you change the port here, you can then create a new Load balancing rule to direct 443 to the pair of FortiGates.

Load balancing rules

These rules also use PAT, but rather than being directed at a specific host, they are directed at a collection of virtual machines called a backend pool. In this case, the pool consists of FortiGate-A and FortiGate-B. These rules are necessary to provide high availability and load balancing for any given service. Referencing the above example – after you have freed up port 443, you would create a new Load balancing rule, configured on port 443 and directed to the FortiGate backend pool.

FortiGate Configuration

*Avoid Clustering*

The architecture described here is given to provide a highly available secure solution within a networking environment that doesn't support traditional HA mechanisms.  If you enable HA features in the FortiGate GUI, you will likely lose connectivity and need to redeploy.  See below in the comments for additional dialogue on the various reasons for this.

Licenses

If you choose PAYG during deployment, you will start with fully licensed FortiGates that are billed directly by Microsoft.  The hourly prices are available in the marketplace description.

However, if you choose BYOL (Bring your own license), the first step of configuration is to install a license. Connect to the web-based management interface at the public IP address assigned to the Azure Load Balancer. This interface will be available via port 443 - each FortiGate has NAT rules for ports 22 and 443 configured through the load balancer.  The first public IP which you selected (or created) will direct to FortiGate-A and the second will direct to FortiGate-B.  If desired, you can modify the ports used through the load balancer configuration in the Azure portal.

Once connected, you will be prompted to install a license file. After you have uploaded the license file, wait for the FortiGate to reboot and connect to the FortiGuard services. Full FortiGuard synchronization can take up to 30 minutes. However, you should be able to connect and continue configuration within about 5 minutes.

Note: The BYOL Marketplace template does not come with a license. In order to obtain a license, you will need to work with your Fortinet representative or network security partner. Alternately, you can email azure@fortinet.com

Management Ports

If you would like to change the management ports in order to allow those ports to be forwarded to internal resources, you will need to change both on the FortiGate (select System -> Admin -> Settings and adjust accordingly), and on the load balancer NAT rules configuration in the Azure portal.

Outbound Communication

In order to allow outbound communication from hosts on the Protected Network to the internet or other external hosts, you will need to configure a policy:

1. Select “Policy & Objects” along the left hand side of the management interface
2. Select “Policy” and “IPv4”
3. Click the “Create New” button in the top tool bar
4. Select Port2 for “Incoming Interface”
5. For Source address you can be as granular as you like. In this example, we’ll use “all”
6. Select Port1 for “Outgoing Interface”
7. For Destination address select “all” – again you can be as granular as you like here
8. For Service select “ALL”
9. Ensure that NAT is enabled
10. Select your desired Security Profiles
11. Click “Ok” at the bottom

For additional information on granular configuration, security profiles, etc., please see the FortiOS Handbook: http://docs.fortinet.com/d/fortigate-fortios-handbook-the-complete-guide-to-fortios-5.2

Inbound Communication

To enable traffic coming from the internet, you will need to configure PAT on the FortiGates. The first step will be to create a Virtual IP:

1. Select “Policy & Objects” along the left hand side of the management interface
2. Select “Objects” and “Virtual IPs”
3. Click the “Create New” button in the top tool bar
4. Type a name. In my example, I’m using Web-HTTPS
5. Select port1 under “Interface”
6. Use the public IP address associated with the Azure load balancer for the External IP Address/Range (type it twice).
7. For the “Mapped IP Address/Range,” use the IP address of your internal host (again type it twice).
8. Select the checkbox next to “Port Forwarding”
9. Select the Protocol you wish to use
10. Type in the port you wish to use. This can be a range or a single port. In the example I’m using 443. If you wish to forward the external port 443, you will need to change the management port of FortiGate-A and the Inbound NAT Rule (both processes are described above). The external port can be mapped to a different internal port here if desired.
11. Click “Ok” at the bottom

Once you have the Virtual IP configured, you need to create a new policy:

1. Select “Policy & Objects” along the left hand side of the management interface
2. Select “Policy” and “IPv4”
3. Click the “Create New” button in the top tool bar
4. Select Port1 for “Incoming Interface”
5. For Source address you can be as granular as you like. In this example, we’ll use “all”
6. Select Port2 for “Outgoing Interface”
7. For Destination address select the name of the Virtual IP that you created
8. For Service select “ALL”
9. Ensure that NAT is enabled
10. Select your desired Security Profiles
11. Click “Ok” at the bottom

Finally, configure a new load balancer rule on the Azure load balancer.  This can be done through the Azure portal, powershell, Azure CLI, or even REST APIs.  However, you do so, be sure to enable floating IP/Direct server return on the load balancer rule.  This enables the load balancer to forward the traffic without changing the destination IP address (so the FortiGate VIP will match traffic destined to the original public IP destination).  This has the benefit of allowing you to reuse the same port (example TCP 443) with multiple VIPs.  It also can be used with FGSP to synchronize state between the two FortiGates in certain asymmetric configurations.

Routing

Through the use of the Azure Load Balancer and the source NAT on incoming traffic to the FortiGates (described above), we are able to achieve high availability for incoming connections. For many common services this is adequate. However, for services requiring the ability to create outbound connections like SMTP servers or Web servers which communicate with other databases, etc., there’s an additional mechanism that needs to be deployed.

In order to force internal->external traffic to route through the FortiGate, we use an Azure feature called User Defined Routes (UDRs). This allows us to specify an alternative to the default Azure router, but it only allows a single router per route. If that router is not available, the traffic gets dropped. Thus, to support highly available internal->external connections we need to programatically change that UDR. There are various ways to do this. 

--Note--

Azure has a recommended deployment option for this.  However, it's not widely used and by all accounts both difficult and expensive.  Here are details on that.

https://channel9.msdn.com/Shows/Azure-Friday/Deploying-Network-Virtual-Appliances-for-High-Availabil...

https://github.com/mspnp/ha-nva

Also, Azure now supports internal load balancers as UDR next hops.  So, depending on your deployment adding an internal load balancer may be preferable for providing outbound and east/west HA.  It has faster failover time (max 10 seconds vs. minimum 30 seconds), and can be used to scale FortiGates horizontally.  Currently, the internal load balancer, while it functions at layer 3, must be configured on a per UDP/TCP port at layer 4.  Thus, any solution which leverages a dual load balancer, must use source-NAT on the FortiGate for all traffic.

Here is the oldest and most widely deployed solution:

The fastest method utilizes an in-VNet virtual server to act as a Software Defined Network (SDN) controller.  It does so by running a monitor script and changing the Azure UDR in the case that FortiGate-A becomes inaccessible.  This controller needs to have access to the internet regardless of the status of either FortiGate, so it is typically placed int the 'outside' subnet.  The testing and PoCs that we have done have been on A0 Ubuntu VMs.  The sample script attached to this post is a Unix shell script that leverages the Azure CLI.  This is provided as a sample and will need to be customized to fit your environment.  You could also create a powershell script and run the same process from a Windows VM (or do something similar from an Azure runbook).

Prior to running the script on an Ubuntu server, you will need to complete the following:

    - Install Azure CLI tools
          sudo apt-get install nodejs-legacy
          sudo apt-get install npm
          sudo npm install -g azure-cli

    - Install JQ
         sudo apt-get install jq

    - Authenticate to Azure. Note: this requires an Azure AD account.  For more information: https://azure.microsoft.com/en-us/documentation/articles/resource-group-create-work-id-from-personal...
         azure login -u
 

   - Set Azure CLI to ARM mode
         azure config mode arm

[AnaFuen]

Hi Martin,

I have a project for FGT-VM in Azure and the customer needs autoscaling. I need to confirm if it is possible that customer initially purchase two FGs single VM or one HA ARM template using BYOL and after the customer can purchase more FGTs with PAYG to add to cluster or the customer will have to purchase others FG-VM BYOL to add more FGTs to cluster?. This is because the client needs to scale the cluster horizontally to increase performance in periods of peak traffic and I need to confirm what options do we have for this scenario.

Regards

Ana Fuentes

[MartTwom]

You can only use a single public load balancer with the FortiGates. However, you can add multiple frontends to that load balancer. Each Frontend will have a unique public IP. And, then, yes you would use D-NAT (in our configuration it’s called Virtual IP) to forward that traffic to an internal web server. I would recommend using the floating IP checkbox in the Azure public load balancer rule configuration. Then, you can create a Virtual IP in the FortiGate config based on the frontend public IP of the Azure load balancer. This way, the configuration can be identical between multiple FortiGates on the load balancer backend.

Regards,
Martin

________________________________


*** Please note that this message and any attachments may contain confidential and proprietary material and information and are intended only for the use of the intended recipient(s). If you are not the intended recipient, you are hereby notified that any review, use, disclosure, dissemination, distribution or copying of this message and any attachments is strictly prohibited. If you have received this email in error, please immediately notify the sender and destroy this e-mail and any attachments and all copies, whether electronic or printed. Please also note that any views, opinions, conclusions or commitments expressed in this message are those of the individual sender and do not necessarily reflect the views of Fortinet, Inc., its affiliates, and emails are not binding on Fortinet and only a writing manually signed by Fortinet's General Counsel can be a binding commitment of Fortinet to Fortinet's customers or partners. Thank you. ***