FortiGate Scalable HA Architecture as Defined in Azure Marketplace

[MartTwom]

Components:

Azure Load Balancer – Abstracted Azure resource which is scalable and resilient. Dynamically splits traffic between the two FortiGates.

Virtual Network – 10.40.0.0/16, also known as VNET

Public Facing Network – 10.40.1.0/24

Protected Network – 10.40.2.0/24

Availability Set – Method of grouping resources within Azure to ensure that they are hosted on separate physical hardware in order to ensure that at any given time (even during upgrades and maintenance) at least one of the set will remain up.

FortiGate – Azure certified virtual appliance running the same OS which is used on our hardware appliances. These will be referenced as FortiGate-A and FortiGate-B.

FortiManager – Dedicated policy and configuration manager, used to keep the configuration in sync between the two FortiGates.

How to deploy:

Utilize the FortiGate HA template which is available in the Microsoft Azure Marketplace to deploy the network resources and FortiGates as depicted in the diagram.

Configuration:

Azure Load Balancer

All traffic coming from outside of Azure will pass through the load balancer first. The load balancer uses Network Address Translation and Port Address Translation (NAT/PAT) to connect a single public IP address to the Azure VNET. Within the Azure portal there are two options for configuring these NAT rules. The first is called “Inbound NAT rules.” The second is termed “Load balancing Rules”

Inbound NAT rules

These rules are applied to a specific host and are not load balanced. As such, these are typically used for management. The template uses ports 443 and 22 for management of FortiGate-A. Ports 8443 and 8022 are similarly directed at FortiGate-B. Once the FortiGates are configured, you can change these ports. For example, if you want to use port 443 for internal web services, you could configure an alternate port on FortiGate-A for management, and modify this rule to use that new port. Once you change the port here, you can then create a new Load balancing rule to direct 443 to the pair of FortiGates.

Load balancing rules

These rules also use PAT, but rather than being directed at a specific host, they are directed at a collection of virtual machines called a backend pool. In this case, the pool consists of FortiGate-A and FortiGate-B. These rules are necessary to provide high availability and load balancing for any given service. Referencing the above example – after you have freed up port 443, you would create a new Load balancing rule, configured on port 443 and directed to the FortiGate backend pool.

FortiGate Configuration

*Avoid Clustering*

The architecture described here is given to provide a highly available secure solution within a networking environment that doesn't support traditional HA mechanisms.  If you enable HA features in the FortiGate GUI, you will likely lose connectivity and need to redeploy.  See below in the comments for additional dialogue on the various reasons for this.

Licenses

If you choose PAYG during deployment, you will start with fully licensed FortiGates that are billed directly by Microsoft.  The hourly prices are available in the marketplace description.

However, if you choose BYOL (Bring your own license), the first step of configuration is to install a license. Connect to the web-based management interface at the public IP address assigned to the Azure Load Balancer. This interface will be available via port 443 - each FortiGate has NAT rules for ports 22 and 443 configured through the load balancer.  The first public IP which you selected (or created) will direct to FortiGate-A and the second will direct to FortiGate-B.  If desired, you can modify the ports used through the load balancer configuration in the Azure portal.

Once connected, you will be prompted to install a license file. After you have uploaded the license file, wait for the FortiGate to reboot and connect to the FortiGuard services. Full FortiGuard synchronization can take up to 30 minutes. However, you should be able to connect and continue configuration within about 5 minutes.

Note: The BYOL Marketplace template does not come with a license. In order to obtain a license, you will need to work with your Fortinet representative or network security partner. Alternately, you can email azure@fortinet.com

Management Ports

If you would like to change the management ports in order to allow those ports to be forwarded to internal resources, you will need to change both on the FortiGate (select System -> Admin -> Settings and adjust accordingly), and on the load balancer NAT rules configuration in the Azure portal.

Outbound Communication

In order to allow outbound communication from hosts on the Protected Network to the internet or other external hosts, you will need to configure a policy:

1. Select “Policy & Objects” along the left hand side of the management interface
2. Select “Policy” and “IPv4”
3. Click the “Create New” button in the top tool bar
4. Select Port2 for “Incoming Interface”
5. For Source address you can be as granular as you like. In this example, we’ll use “all”
6. Select Port1 for “Outgoing Interface”
7. For Destination address select “all” – again you can be as granular as you like here
8. For Service select “ALL”
9. Ensure that NAT is enabled
10. Select your desired Security Profiles
11. Click “Ok” at the bottom

For additional information on granular configuration, security profiles, etc., please see the FortiOS Handbook: http://docs.fortinet.com/d/fortigate-fortios-handbook-the-complete-guide-to-fortios-5.2

Inbound Communication

To enable traffic coming from the internet, you will need to configure PAT on the FortiGates. The first step will be to create a Virtual IP:

1. Select “Policy & Objects” along the left hand side of the management interface
2. Select “Objects” and “Virtual IPs”
3. Click the “Create New” button in the top tool bar
4. Type a name. In my example, I’m using Web-HTTPS
5. Select port1 under “Interface”
6. Use the public IP address associated with the Azure load balancer for the External IP Address/Range (type it twice).
7. For the “Mapped IP Address/Range,” use the IP address of your internal host (again type it twice).
8. Select the checkbox next to “Port Forwarding”
9. Select the Protocol you wish to use
10. Type in the port you wish to use. This can be a range or a single port. In the example I’m using 443. If you wish to forward the external port 443, you will need to change the management port of FortiGate-A and the Inbound NAT Rule (both processes are described above). The external port can be mapped to a different internal port here if desired.
11. Click “Ok” at the bottom

Once you have the Virtual IP configured, you need to create a new policy:

1. Select “Policy & Objects” along the left hand side of the management interface
2. Select “Policy” and “IPv4”
3. Click the “Create New” button in the top tool bar
4. Select Port1 for “Incoming Interface”
5. For Source address you can be as granular as you like. In this example, we’ll use “all”
6. Select Port2 for “Outgoing Interface”
7. For Destination address select the name of the Virtual IP that you created
8. For Service select “ALL”
9. Ensure that NAT is enabled
10. Select your desired Security Profiles
11. Click “Ok” at the bottom

Finally, configure a new load balancer rule on the Azure load balancer.  This can be done through the Azure portal, powershell, Azure CLI, or even REST APIs.  However, you do so, be sure to enable floating IP/Direct server return on the load balancer rule.  This enables the load balancer to forward the traffic without changing the destination IP address (so the FortiGate VIP will match traffic destined to the original public IP destination).  This has the benefit of allowing you to reuse the same port (example TCP 443) with multiple VIPs.  It also can be used with FGSP to synchronize state between the two FortiGates in certain asymmetric configurations.

Routing

Through the use of the Azure Load Balancer and the source NAT on incoming traffic to the FortiGates (described above), we are able to achieve high availability for incoming connections. For many common services this is adequate. However, for services requiring the ability to create outbound connections like SMTP servers or Web servers which communicate with other databases, etc., there’s an additional mechanism that needs to be deployed.

In order to force internal->external traffic to route through the FortiGate, we use an Azure feature called User Defined Routes (UDRs). This allows us to specify an alternative to the default Azure router, but it only allows a single router per route. If that router is not available, the traffic gets dropped. Thus, to support highly available internal->external connections we need to programatically change that UDR. There are various ways to do this. 

--Note--

Azure has a recommended deployment option for this.  However, it's not widely used and by all accounts both difficult and expensive.  Here are details on that.

https://channel9.msdn.com/Shows/Azure-Friday/Deploying-Network-Virtual-Appliances-for-High-Availabil...

https://github.com/mspnp/ha-nva

Also, Azure now supports internal load balancers as UDR next hops.  So, depending on your deployment adding an internal load balancer may be preferable for providing outbound and east/west HA.  It has faster failover time (max 10 seconds vs. minimum 30 seconds), and can be used to scale FortiGates horizontally.  Currently, the internal load balancer, while it functions at layer 3, must be configured on a per UDP/TCP port at layer 4.  Thus, any solution which leverages a dual load balancer, must use source-NAT on the FortiGate for all traffic.

Here is the oldest and most widely deployed solution:

The fastest method utilizes an in-VNet virtual server to act as a Software Defined Network (SDN) controller.  It does so by running a monitor script and changing the Azure UDR in the case that FortiGate-A becomes inaccessible.  This controller needs to have access to the internet regardless of the status of either FortiGate, so it is typically placed int the 'outside' subnet.  The testing and PoCs that we have done have been on A0 Ubuntu VMs.  The sample script attached to this post is a Unix shell script that leverages the Azure CLI.  This is provided as a sample and will need to be customized to fit your environment.  You could also create a powershell script and run the same process from a Windows VM (or do something similar from an Azure runbook).

Prior to running the script on an Ubuntu server, you will need to complete the following:

    - Install Azure CLI tools
          sudo apt-get install nodejs-legacy
          sudo apt-get install npm
          sudo npm install -g azure-cli

    - Install JQ
         sudo apt-get install jq

    - Authenticate to Azure. Note: this requires an Azure AD account.  For more information: https://azure.microsoft.com/en-us/documentation/articles/resource-group-create-work-id-from-personal...
         azure login -u
 

   - Set Azure CLI to ARM mode
         azure config mode arm

[EricBocl]

In the Fortigate VM for Azure, there is an account auto-created during the installation of the environment: the  admin account.

I do not have the password for this account.

It has super_admin profile.

I think it's used by the Azure load balancer to check availability.

If I put a trusted host for this account, the Azure load balancer can not see my published services.

So, this account is necessary for the probe SSH.

Who knows the password? Microsoft? Fortinet?

How can we be sure that this admin account will not be used by hackers?

Is there a way to secure it?

I changed the port for the probe SSH.

Thanks

[MartTwom]

Hi Eric,

Good question.  The admin account is not used by the Azure load balancer. It's created because the FortiGate VM requires it. However, "admin" is on the non-approved list for the Azure marketplace, so it can't be directly configured via template paramters.  Instead, it's created with a password which is randomly assigned at deployment.  I do recommend that you either disable the account or change the password.  You can only change the admin password via the CLI:

config system admin
  edit "admin"
  set password "new-password"
next
end

Regarding the load balancer probe: If you stick with the configured (SSH) probe, you will need to ensure that your FortiGate virtual appliance will respond.  Interestingly, this probe comes from an IP outside of the network (if I recall correctly, it's actually a public IP).  Of course, you can change the probe to something other than SSH if you prefer.

[PeteVuko]

Martin,

What is the minimum requirement to this HA architecture to work? I mean in terms of VM license and Azure compute instance.

VM02 on D3v2?

Rgrds,

Peter

[MartTwom]

Hi Peter, the minimum instance type that we support in Azure is D2v2 (which requires a FortiGate VM02-AZ license).  This will work in the above HA architectures. 

*Note: The VM02 license will not work with the D3v2 instance - that would require a VM04 license.

[andrewwilson_FTNT]

Hi Martin. Great info here. I have a quick question around the Azure LB. What Probe does it use to detect if one of the FG's is down in the pool? I can't see any info on Fuse relating to what probes we support in Azure.

[Jeroen_Bismans_FTNT]

Hi Andrew

If you roll out the default HA image from the Azure Marketplace, the health probes are on TCP 22.
Azure supports health probes on HTTP and any TCP port so you should be able to choose any TCP port we listen on.

Hope that helps.
Martin, feel free to correct me if I'm wrong. ;)

In Reply to Andrew Wilson:

Hi Martin. Great info here. I have a quick question around the Azure LB. What Probe does it use to detect if one of the FG's is down in the pool? I can't see any info on Fuse relating to what probes we support in Azure.

[MartHofm]

Hi,

did anyone had success with the the ha-nva docker solution from MS? My clientMonitor Docker containers won´t come up, they throwing error

2017-06-28 18:12:35,984 ERROR [main:Main@80] - Error running daemon
org.apache.commons.daemon.DaemonInitException: Error processing command line arguments
        at com.microsoft.azure.practices.nvadaemon.NvaDaemon.init(NvaDaemon.java:62)
        at com.microsoft.azure.practices.nvadaemon.Main.runDaemon(Main.java:22)
        at com.microsoft.azure.practices.nvadaemon.Main.main(Main.java:78)
2017-06-28 18:12:35,990 INFO  [Thread-0:Main@52] - JVM is shutting down.

nvadaemon-remote.json is right formatted...

greetings

Martin

[Mir_Mirg]

HI All

Has anyone successfully deployed mspnp/ha-nva solution ?

[Mir_Mirg]

I successfully setup this solution only by replacing Azure LB by a redundant pair of F5 BIPIPs. works like a charm :)

[AndryanVT]

Hello Martin,

Thanks for the documentations

If I want to route multiple web servers through the FortiGate, technically I can make LB for each webservers and load balance to the Fortigate. Then we configured FortiGate with D-NAT towards the IP Private of each Web Server right?

" However, you do so, be sure to enable floating IP/Direct server return on the load balancer rule.  This enables the load balancer to forward the traffic without changing the destination IP address (so the FortiGate VIP will match traffic destined to the original public IP destination). "

And then, do you have any documentation like this for deployment using Azure Internal Load balancer as next hop for the UDR? 

Thanks

Andryan VT