FortiGate Scalable HA Architecture as Defined in Azure Marketplace

MartTwom · ‎02-20-2016

Components:

Azure Load Balancer – Abstracted Azure resource which is scalable and resilient. Dynamically splits traffic between the two FortiGates.

Virtual Network – 10.40.0.0/16, also known as VNET

Public Facing Network – 10.40.1.0/24

Protected Network – 10.40.2.0/24

Availability Set – Method of grouping resources within Azure to ensure that they are hosted on separate physical hardware in order to ensure that at any given time (even during upgrades and maintenance) at least one of the set will remain up.

FortiGate – Azure certified virtual appliance running the same OS which is used on our hardware appliances. These will be referenced as FortiGate-A and FortiGate-B.

FortiManager – Dedicated policy and configuration manager, used to keep the configuration in sync between the two FortiGates.

How to deploy:

Utilize the FortiGate HA template which is available in the Microsoft Azure Marketplace to deploy the network resources and FortiGates as depicted in the diagram.

Configuration:

Azure Load Balancer

All traffic coming from outside of Azure will pass through the load balancer first. The load balancer uses Network Address Translation and Port Address Translation (NAT/PAT) to connect a single public IP address to the Azure VNET. Within the Azure portal there are two options for configuring these NAT rules. The first is called “Inbound NAT rules.” The second is termed “Load balancing Rules”

Inbound NAT rules

These rules are applied to a specific host and are not load balanced. As such, these are typically used for management. The template uses ports 443 and 22 for management of FortiGate-A. Ports 8443 and 8022 are similarly directed at FortiGate-B. Once the FortiGates are configured, you can change these ports. For example, if you want to use port 443 for internal web services, you could configure an alternate port on FortiGate-A for management, and modify this rule to use that new port. Once you change the port here, you can then create a new Load balancing rule to direct 443 to the pair of FortiGates.

Load balancing rules

These rules also use PAT, but rather than being directed at a specific host, they are directed at a collection of virtual machines called a backend pool. In this case, the pool consists of FortiGate-A and FortiGate-B. These rules are necessary to provide high availability and load balancing for any given service. Referencing the above example – after you have freed up port 443, you would create a new Load balancing rule, configured on port 443 and directed to the FortiGate backend pool.

FortiGate Configuration

Avoid Clustering

The architecture described here is given to provide a highly available secure solution within a networking environment that doesn't support traditional HA mechanisms. If you enable HA features in the FortiGate GUI, you will likely lose connectivity and need to redeploy. See below in the comments for additional dialogue on the various reasons for this.

Licenses

If you choose PAYG during deployment, you will start with fully licensed FortiGates that are billed directly by Microsoft. The hourly prices are available in the marketplace description.

However, if you choose BYOL (Bring your own license), the first step of configuration is to install a license. Connect to the web-based management interface at the public IP address assigned to the Azure Load Balancer. This interface will be available via port 443 - each FortiGate has NAT rules for ports 22 and 443 configured through the load balancer. The first public IP which you selected (or created) will direct to FortiGate-A and the second will direct to FortiGate-B. If desired, you can modify the ports used through the load balancer configuration in the Azure portal.

Once connected, you will be prompted to install a license file. After you have uploaded the license file, wait for the FortiGate to reboot and connect to the FortiGuard services. Full FortiGuard synchronization can take up to 30 minutes. However, you should be able to connect and continue configuration within about 5 minutes.

Note: The BYOL Marketplace template does not come with a license. In order to obtain a license, you will need to work with your Fortinet representative or network security partner. Alternately, you can email azure@fortinet.com

Management Ports

If you would like to change the management ports in order to allow those ports to be forwarded to internal resources, you will need to change both on the FortiGate (select System -> Admin -> Settings and adjust accordingly), and on the load balancer NAT rules configuration in the Azure portal.

Outbound Communication

In order to allow outbound communication from hosts on the Protected Network to the internet or other external hosts, you will need to configure a policy:

Select “Policy & Objects” along the left hand side of the management interface
Select “Policy” and “IPv4”
Click the “Create New” button in the top tool bar
Select Port2 for “Incoming Interface”
For Source address you can be as granular as you like. In this example, we’ll use “all”
Select Port1 for “Outgoing Interface”
For Destination address select “all” – again you can be as granular as you like here
For Service select “ALL”
Ensure that NAT is enabled
Select your desired Security Profiles
Click “Ok” at the bottom

For additional information on granular configuration, security profiles, etc., please see the FortiOS Handbook: http://docs.fortinet.com/d/fortigate-fortios-handbook-the-complete-guide-to-fortios-5.2

Inbound Communication

To enable traffic coming from the internet, you will need to configure PAT on the FortiGates. The first step will be to create a Virtual IP:

Select “Policy & Objects” along the left hand side of the management interface
Select “Objects” and “Virtual IPs”
Click the “Create New” button in the top tool bar
Type a name. In my example, I’m using Web-HTTPS
Select port1 under “Interface”
Use the public IP address associated with the Azure load balancer for the External IP Address/Range (type it twice).
For the “Mapped IP Address/Range,” use the IP address of your internal host (again type it twice).
Select the checkbox next to “Port Forwarding”
Select the Protocol you wish to use
Type in the port you wish to use. This can be a range or a single port. In the example I’m using 443. If you wish to forward the external port 443, you will need to change the management port of FortiGate-A and the Inbound NAT Rule (both processes are described above). The external port can be mapped to a different internal port here if desired.
Click “Ok” at the bottom

Once you have the Virtual IP configured, you need to create a new policy:

Select “Policy & Objects” along the left hand side of the management interface
Select “Policy” and “IPv4”
Click the “Create New” button in the top tool bar
Select Port1 for “Incoming Interface”
For Source address you can be as granular as you like. In this example, we’ll use “all”
Select Port2 for “Outgoing Interface”
For Destination address select the name of the Virtual IP that you created
For Service select “ALL”
Ensure that NAT is enabled
Select your desired Security Profiles
Click “Ok” at the bottom

Finally, configure a new load balancer rule on the Azure load balancer. This can be done through the Azure portal, powershell, Azure CLI, or even REST APIs. However, you do so, be sure to enable floating IP/Direct server return on the load balancer rule. This enables the load balancer to forward the traffic without changing the destination IP address (so the FortiGate VIP will match traffic destined to the original public IP destination). This has the benefit of allowing you to reuse the same port (example TCP 443) with multiple VIPs. It also can be used with FGSP to synchronize state between the two FortiGates in certain asymmetric configurations.

Routing

Through the use of the Azure Load Balancer and the source NAT on incoming traffic to the FortiGates (described above), we are able to achieve high availability for incoming connections. For many common services this is adequate. However, for services requiring the ability to create outbound connections like SMTP servers or Web servers which communicate with other databases, etc., there’s an additional mechanism that needs to be deployed.

In order to force internal->external traffic to route through the FortiGate, we use an Azure feature called User Defined Routes (UDRs). This allows us to specify an alternative to the default Azure router, but it only allows a single router per route. If that router is not available, the traffic gets dropped. Thus, to support highly available internal->external connections we need to programatically change that UDR. There are various ways to do this.

--Note--

Azure has a recommended deployment option for this. However, it's not widely used and by all accounts both difficult and expensive. Here are details on that.

https://channel9.msdn.com/Shows/Azure-Friday/Deploying-Network-Virtual-Appliances-for-High-Availabil...

https://github.com/mspnp/ha-nva

Also, Azure now supports internal load balancers as UDR next hops. So, depending on your deployment adding an internal load balancer may be preferable for providing outbound and east/west HA. It has faster failover time (max 10 seconds vs. minimum 30 seconds), and can be used to scale FortiGates horizontally. Currently, the internal load balancer, while it functions at layer 3, must be configured on a per UDP/TCP port at layer 4. Thus, any solution which leverages a dual load balancer, must use source-NAT on the FortiGate for all traffic.

Here is the oldest and most widely deployed solution:

The fastest method utilizes an in-VNet virtual server to act as a Software Defined Network (SDN) controller. It does so by running a monitor script and changing the Azure UDR in the case that FortiGate-A becomes inaccessible. This controller needs to have access to the internet regardless of the status of either FortiGate, so it is typically placed int the 'outside' subnet. The testing and PoCs that we have done have been on A0 Ubuntu VMs. The sample script attached to this post is a Unix shell script that leverages the Azure CLI. This is provided as a sample and will need to be customized to fit your environment. You could also create a powershell script and run the same process from a Windows VM (or do something similar from an Azure runbook).

Prior to running the script on an Ubuntu server, you will need to complete the following:

    - Install Azure CLI tools
          sudo apt-get install nodejs-legacy
          sudo apt-get install npm
      sudo npm install -g azure-cli

    - Install JQ
   sudo apt-get install jq

    - Authenticate to Azure. Note: this requires an Azure AD account. For more information: https://azure.microsoft.com/en-us/documentation/articles/resource-group-create-work-id-from-personal...
         azure login -u

- Set Azure CLI to ARM mode
azure config mode arm

psoni_FTNT · ‎10-14-2016

Hi Martin,

Does this mean customer has to use Azure load balancer compulsory to achieve HA of FortiGate? Is it possible to implement HA in same fashion what we do in physical appliances? If possible, can you please share sample configuration?

Regds,

Pratik Soni

MartTwom · ‎10-15-2016

Hi Pratik,

Traditional HA is not currently possible within the public cloud. So, the architecture described in the original post is an option to still provide a highly available or redundant solution within the inherit limitations of Azure.

Here are the key limitations that make traditional HA impossible:
- No access to layer 2 (gratuitous ARP or any ARP manipulation isn't possible).
- No multicast
- No Virtual IP nor IP mobility
- Limited ability to manipulate intraVNet routing.

So, to answer to your first question - Yes, in order to provide a resilient security environment to public, internet-based clients, an Azure load balancer is required. There is an architecture designed for east/west inspection between virtual networks (not north/south - the Azure load balancer is still required for traffic sourced from public IPs) where you can utilize a combination of IPSec, BGP, and FGSP to provide immediate failover and session synchronization. I'm working on some documentation for this and will post a link in here soon.

yguerinel_FTNT · ‎01-05-2017

In Reply to Martin Twombly:

[...]

the Azure load balancer is still required for traffic sourced from public IPs) where you can utilize a combination of IPSec, BGP, and FGSP to provide immediate failover and session synchronization. I'm working on some documentation for this and will post a link in here soon.

Hello Martin,

Great post ! Do you have any news about this doc ? I am currently more and more questioned about "HA" in Azure and looking for as much information as I can get.

Yann

BobbMazz · ‎05-23-2017

Hi Martin,

I was hoping you could assist with the script as the UDR and servers sit in a different RG than the HA Fortinet's. We have relatively complex environment being built out in an NCUS and WEU Azure tenancy. The client will require many site to site VPN's to onprem which leverage a UDR.

Thanks,

-Bobby

LeifHard · ‎05-24-2017

Bobby,

Resource Groups are just organizational container abstract concepts.

The FortiNet can be in it's own RG and other devices in their own and they talk properly as long as they are within the same subscription.

LeifHard · ‎05-24-2017

Attached is a very simplified version of a single FortiNet deployed in Azure. You can see where the DMZ network has it's own NSG and UDR. You can pretty much copy that network as a template and deploy many other networks behind the FortiNet in the same way.

BobbMazz · ‎05-24-2017

Thank you, Leif!

I think my concern was more so around routing, and what options there were to update the UDR for next hop IP. I think we are going to use an internal load balancer with a private IP, and point the UDR to that VM as the next hop.

Thanks,

-Bobby

rickyrickuk · ‎10-18-2016

Martin

In the section below, should the ports on steps 4 and 6 be different?

Once you have the Virtual IP configured, you need to create a new policy:

Select “Policy & Objects” along the left hand side of the management interface
Select “Policy” and “IPv4”
Click the “Create New” button in the top tool bar
Select Port1 for “Incoming Interface”
For Source address you can be as granular as you like. In this example, we’ll use “all”
Select Port1 for “Outgoing Interface”
For Destination address select the name of the Virtual IP that you created
For Service select “ALL”
Ensure that NAT is enabled
Select your desired Security Profiles
Click “Ok” at the bottom

Thanks

Richard

MartTwom · ‎10-18-2016

Hi Richard,

Yes, you are correct. Thank you for catching that!

I'll update it now.

MartTwom · ‎10-18-2016

Q&A

We need to configure one of the Interfaces for HA heart beat, what is the right one to use? How it should be configured from the perspective of Azure Subnets / VNETs? Should it be separate interface or we can just use internal / external one?

No HA heart beat is needed nor will one function in Azure. This particular architecture relies on the Azure load balancer for high availability.

This architecture is recommended to provide a highly available or redundant solution within the inherit limitations of Azure which prohibit most traditional HA mechanisms.
Here are the key limitations that make traditional HA impossible:
- No access to layer 2 (gratuitous ARP or any ARP manipulation isn't possible).
- No multicast
- No Virtual IP nor IP mobility
- Limited ability to manipulate intraVNet routing.

How can we create or move additional interfaces to the external subnet to have more interfaces for service publishing?

This is not currently possible, although some of the preview features that the Azure network team is working on may make this feasible in the next 6 months. At this time, you can use the Azure load balancer to host multiple public IPs, and then use custom ports between the load balancer and the single ‘public’ interface of the FortiGates. The ports only need to be custom for this one hop - externally and behind the FortiGates you can use standard ports.

Do you have any plans to support multiple IPs on the same interface?

Yes. This should work as soon as it’s available in Azure.

Do we have all the listed options relevant for the Azure Deployment: Active /Active and Active / Passive?

No. For inbound connections sourced from public resources (outside of Azure), this is Active/Active. For outbound connections sourced from internal VMs, this is Active/Passive, but only if a method of programmatically altering the UDRs is also enabled.

If we have Active / Active option, does this mean that devices will exchange information about the sessions?

No, in this architecture the FortiGates operate independently. Due to the limitations mentioned above, session sync doesn’t provide any value. Specifically, we can synchronize the sessions, but there is no way to move them from one to the other without requiring reestablishment. There is a separate architecture which does allow session sync for east/west inspection. However, even in that scenario public access into the Virtual Network will not provide session maintenance (this is due to the NAT deployment inherent to public access into Azure).

Is it possible why/why not to put an Internal Load balancer in front of internal endpoints and setup UDR to this ILB instead of the Interface of one of the appliances?

The Azure load balancer is not a layer 3 device and, as such, can’t be configured as a next hop for UDRs. It would be possible to put a Fortinet appliance or third party appliance in front of the FortiGates which could operate at layer 3 and provide this functionality, but that just pushes the problem of redundancy up one level.

If we are publishing a service through load balancing rules and then they go to VIP on the FortiGate. As VIP from the IP perspective is limited to the IP of the FortiGate interface, the only difference between services will be NAT-ed port. So do we need to create VIP for every published service?

Yes, this is correct.

Are all the options listed in the FortiGate GUI supported?

Yes and No. All functions are supported by Fortinet for our VMs, but, as we’ve discussed not all options will work in every scenario. Many of the options work in Azure only with VPN-based architectures. Some functions, such as the ability to utilize more than one IP per interface are slowly being enabled on the Azure side. One function that we don’t expect to ever work in Azure is VLANs.

What will be the procedure for the OS / firmware updates for the appliances?

The procedure is identical to our physical or other virtual appliance. These updates are downloaded from our support site (support.fortinet.com), and then uploaded to the FortiGate (typically via the GUI).

Should the FortiGate have routes to all subnets and VNets it is protecting specified in the Routing Table?

Yes, absolutely. The FortiGate is a layer 3 device and needs a route for any network with which it is expected to communicate. Azure provides a default route out port1 and routes are automatically populated for any connected subnets. Beyond that, if you don’t want traffic to go out port1, then you need to provide more specific routes. The next hop (Gateway) for these routes should be the first IP address of the subnet on which the FortiGate network interface resides. For example if port2 is configured with 10.0.2.10/24, and you want to provide a route via port2 for a new subnet with an address space of 10.0.3.0/24, then the Gateway would be 10.0.2.1. This IP address belongs to an automatically created and maintained Azure router that has an interface in every subnet. On a side note: when you create a UDR and assign it to a subnet, you are essentially adding a policy route to this Azure router.

Do you have advice on how to handle routing between VNets which are connected via ExpressRoute?

I would recommend checking with Microsoft on this. UDRs can be applied to the GatewaySubnet, but in some cases they are ignored. I have not seen documentation on this, so I’m not sure what the expected behavior is.

Is there any other documentation available that specifically addresses FortiGates in Azure?

Any information that we have published is referenced on our Azure website: http://www.fortinet.com/azure