FortiGate Scalable HA Architecture as Defined in Azure Marketplace

[MartTwom]

Components:

Azure Load Balancer – Abstracted Azure resource which is scalable and resilient. Dynamically splits traffic between the two FortiGates.

Virtual Network – 10.40.0.0/16, also known as VNET

Public Facing Network – 10.40.1.0/24

Protected Network – 10.40.2.0/24

Availability Set – Method of grouping resources within Azure to ensure that they are hosted on separate physical hardware in order to ensure that at any given time (even during upgrades and maintenance) at least one of the set will remain up.

FortiGate – Azure certified virtual appliance running the same OS which is used on our hardware appliances. These will be referenced as FortiGate-A and FortiGate-B.

FortiManager – Dedicated policy and configuration manager, used to keep the configuration in sync between the two FortiGates.

How to deploy:

Utilize the FortiGate HA template which is available in the Microsoft Azure Marketplace to deploy the network resources and FortiGates as depicted in the diagram.

Configuration:

Azure Load Balancer

All traffic coming from outside of Azure will pass through the load balancer first. The load balancer uses Network Address Translation and Port Address Translation (NAT/PAT) to connect a single public IP address to the Azure VNET. Within the Azure portal there are two options for configuring these NAT rules. The first is called “Inbound NAT rules.” The second is termed “Load balancing Rules”

Inbound NAT rules

These rules are applied to a specific host and are not load balanced. As such, these are typically used for management. The template uses ports 443 and 22 for management of FortiGate-A. Ports 8443 and 8022 are similarly directed at FortiGate-B. Once the FortiGates are configured, you can change these ports. For example, if you want to use port 443 for internal web services, you could configure an alternate port on FortiGate-A for management, and modify this rule to use that new port. Once you change the port here, you can then create a new Load balancing rule to direct 443 to the pair of FortiGates.

Load balancing rules

These rules also use PAT, but rather than being directed at a specific host, they are directed at a collection of virtual machines called a backend pool. In this case, the pool consists of FortiGate-A and FortiGate-B. These rules are necessary to provide high availability and load balancing for any given service. Referencing the above example – after you have freed up port 443, you would create a new Load balancing rule, configured on port 443 and directed to the FortiGate backend pool.

FortiGate Configuration

*Avoid Clustering*

The architecture described here is given to provide a highly available secure solution within a networking environment that doesn't support traditional HA mechanisms.  If you enable HA features in the FortiGate GUI, you will likely lose connectivity and need to redeploy.  See below in the comments for additional dialogue on the various reasons for this.

Licenses

If you choose PAYG during deployment, you will start with fully licensed FortiGates that are billed directly by Microsoft.  The hourly prices are available in the marketplace description.

However, if you choose BYOL (Bring your own license), the first step of configuration is to install a license. Connect to the web-based management interface at the public IP address assigned to the Azure Load Balancer. This interface will be available via port 443 - each FortiGate has NAT rules for ports 22 and 443 configured through the load balancer.  The first public IP which you selected (or created) will direct to FortiGate-A and the second will direct to FortiGate-B.  If desired, you can modify the ports used through the load balancer configuration in the Azure portal.

Once connected, you will be prompted to install a license file. After you have uploaded the license file, wait for the FortiGate to reboot and connect to the FortiGuard services. Full FortiGuard synchronization can take up to 30 minutes. However, you should be able to connect and continue configuration within about 5 minutes.

Note: The BYOL Marketplace template does not come with a license. In order to obtain a license, you will need to work with your Fortinet representative or network security partner. Alternately, you can email azure@fortinet.com

Management Ports

If you would like to change the management ports in order to allow those ports to be forwarded to internal resources, you will need to change both on the FortiGate (select System -> Admin -> Settings and adjust accordingly), and on the load balancer NAT rules configuration in the Azure portal.

Outbound Communication

In order to allow outbound communication from hosts on the Protected Network to the internet or other external hosts, you will need to configure a policy:

1. Select “Policy & Objects” along the left hand side of the management interface
2. Select “Policy” and “IPv4”
3. Click the “Create New” button in the top tool bar
4. Select Port2 for “Incoming Interface”
5. For Source address you can be as granular as you like. In this example, we’ll use “all”
6. Select Port1 for “Outgoing Interface”
7. For Destination address select “all” – again you can be as granular as you like here
8. For Service select “ALL”
9. Ensure that NAT is enabled
10. Select your desired Security Profiles
11. Click “Ok” at the bottom

For additional information on granular configuration, security profiles, etc., please see the FortiOS Handbook: http://docs.fortinet.com/d/fortigate-fortios-handbook-the-complete-guide-to-fortios-5.2

Inbound Communication

To enable traffic coming from the internet, you will need to configure PAT on the FortiGates. The first step will be to create a Virtual IP:

1. Select “Policy & Objects” along the left hand side of the management interface
2. Select “Objects” and “Virtual IPs”
3. Click the “Create New” button in the top tool bar
4. Type a name. In my example, I’m using Web-HTTPS
5. Select port1 under “Interface”
6. Use the public IP address associated with the Azure load balancer for the External IP Address/Range (type it twice).
7. For the “Mapped IP Address/Range,” use the IP address of your internal host (again type it twice).
8. Select the checkbox next to “Port Forwarding”
9. Select the Protocol you wish to use
10. Type in the port you wish to use. This can be a range or a single port. In the example I’m using 443. If you wish to forward the external port 443, you will need to change the management port of FortiGate-A and the Inbound NAT Rule (both processes are described above). The external port can be mapped to a different internal port here if desired.
11. Click “Ok” at the bottom

Once you have the Virtual IP configured, you need to create a new policy:

1. Select “Policy & Objects” along the left hand side of the management interface
2. Select “Policy” and “IPv4”
3. Click the “Create New” button in the top tool bar
4. Select Port1 for “Incoming Interface”
5. For Source address you can be as granular as you like. In this example, we’ll use “all”
6. Select Port2 for “Outgoing Interface”
7. For Destination address select the name of the Virtual IP that you created
8. For Service select “ALL”
9. Ensure that NAT is enabled
10. Select your desired Security Profiles
11. Click “Ok” at the bottom

Finally, configure a new load balancer rule on the Azure load balancer.  This can be done through the Azure portal, powershell, Azure CLI, or even REST APIs.  However, you do so, be sure to enable floating IP/Direct server return on the load balancer rule.  This enables the load balancer to forward the traffic without changing the destination IP address (so the FortiGate VIP will match traffic destined to the original public IP destination).  This has the benefit of allowing you to reuse the same port (example TCP 443) with multiple VIPs.  It also can be used with FGSP to synchronize state between the two FortiGates in certain asymmetric configurations.

Routing

Through the use of the Azure Load Balancer and the source NAT on incoming traffic to the FortiGates (described above), we are able to achieve high availability for incoming connections. For many common services this is adequate. However, for services requiring the ability to create outbound connections like SMTP servers or Web servers which communicate with other databases, etc., there’s an additional mechanism that needs to be deployed.

In order to force internal->external traffic to route through the FortiGate, we use an Azure feature called User Defined Routes (UDRs). This allows us to specify an alternative to the default Azure router, but it only allows a single router per route. If that router is not available, the traffic gets dropped. Thus, to support highly available internal->external connections we need to programatically change that UDR. There are various ways to do this. 

--Note--

Azure has a recommended deployment option for this.  However, it's not widely used and by all accounts both difficult and expensive.  Here are details on that.

https://channel9.msdn.com/Shows/Azure-Friday/Deploying-Network-Virtual-Appliances-for-High-Availabil...

https://github.com/mspnp/ha-nva

Also, Azure now supports internal load balancers as UDR next hops.  So, depending on your deployment adding an internal load balancer may be preferable for providing outbound and east/west HA.  It has faster failover time (max 10 seconds vs. minimum 30 seconds), and can be used to scale FortiGates horizontally.  Currently, the internal load balancer, while it functions at layer 3, must be configured on a per UDP/TCP port at layer 4.  Thus, any solution which leverages a dual load balancer, must use source-NAT on the FortiGate for all traffic.

Here is the oldest and most widely deployed solution:

The fastest method utilizes an in-VNet virtual server to act as a Software Defined Network (SDN) controller.  It does so by running a monitor script and changing the Azure UDR in the case that FortiGate-A becomes inaccessible.  This controller needs to have access to the internet regardless of the status of either FortiGate, so it is typically placed int the 'outside' subnet.  The testing and PoCs that we have done have been on A0 Ubuntu VMs.  The sample script attached to this post is a Unix shell script that leverages the Azure CLI.  This is provided as a sample and will need to be customized to fit your environment.  You could also create a powershell script and run the same process from a Windows VM (or do something similar from an Azure runbook).

Prior to running the script on an Ubuntu server, you will need to complete the following:

    - Install Azure CLI tools
          sudo apt-get install nodejs-legacy
          sudo apt-get install npm
          sudo npm install -g azure-cli

    - Install JQ
         sudo apt-get install jq

    - Authenticate to Azure. Note: this requires an Azure AD account.  For more information: https://azure.microsoft.com/en-us/documentation/articles/resource-group-create-work-id-from-personal...
         azure login -u
 

   - Set Azure CLI to ARM mode
         azure config mode arm

[MarkAskh]

Hi Martin,

Great information, I was hoping you could however elaborate a little more on enabling HA for the FGT nodes as I was unable to find this in your post. I currently am trying to enable HA and having issues where the FGT appliance drops off the network and is not recoverable due to the lack of a console in the Azure world. Fortinet support are unable to help so I am reaching out to you for advice.

Best Rgds

Mark

Infrastructure Architect

Fred IT

[MartTwom]

Hi Mark,

Good question.  I apologize for not makeing it clear in the document.  This architecture is designed as a replacement for traditional HA protocols.  Within Azure, it's not possible do any layer 2 manipulation.  Nor, can you move IP addresses from one VM to another while either VM is live.  This makes traditional HA and layer 3 redundancy protocols impossible to use.  So, what you're seeing when you try to use HA protocols is expected behavior.  For many deployments this architecture provides the necessary redundancy and failover to achieve desired high availability.  If it's not adequate for your application(s), please let us know, and escalate to Microsoft as well.

Regards,

Martin

[simontzeonyu_FTNT]

Hi Martin,

Do we have information for example setup of Geo-Load/site Balancing with the Azure Traffic manager? Or it is straight foward use Azure traffic manager for DNS-Level load balancing like Web application server.

https://azure.microsoft.com/en-us/documentation/articles/virtual-machines-linux-load-balance/

Thanks and regards,

Simon

[MartTwom]

Hi Simon, not yet.  So far most interest has been redundancy within a Virtual Network.  However, I have configured Traffic Manager myself and was impressed with how simple it was to setup.

[MartTwom]

Logging in to Azure CLI with a Certificate

For the Azure SDN Controller/Monitor Station, it is better to login with a certificate, as this can be scripted and continue to function after reboots/timeouts.

Here's a link to the Azure documentation for how to create a service principal:
https://azure.microsoft.com/en-us/documentation/articles/resource-group-authenticate-service-princip...

Here are the commands and notes that I used to accomplish this:

azure config mode arm

#You will need to login with a username in order to set this up.  Once, you have the service principal, you can logout and back in with that.

azure login -u

#Create a certificate

openssl req -x509 -days 3650 -newkey rsa:2048 -out cert.pem -nodes -subj '/CN=exampleapp'

#Combine the public and private keys
cat privkey.pem cert.pem > examplecert.pem

#Certificate data is the contents of the public certificate - between (and not including) ----BEGIN CERTIFICATE---- and ----END CERTIFICATE----

## I'm not actually using a specific app nor website, so these can be arbitrary --err descriptive

### Grab the Object Id: from the output of the sp create command

azure ad sp create -n "twomblyexampleapp" --home-page "https://fortidemo.xyz" -i "https://fortidemo.xyz/example" --key-value "Certificate data"

#Get Subscription ID

azure account list

#Give your new service principal a role (ie. permission to modify route table assignments in Azure)
## I used Contributor, but you could use a custom role if you want to narrow the permission set for security purposes.

azure role assignment create --objectId

#Get certificate thumbprint

openssl x509 -in examplecert.pem -fingerprint -noout | sed 's/SHA1 Fingerprint=//g'  | sed 's/://g'

#Get App ID

azure ad app show --search twomblyexampleapp --json | jq -r '.[0].appId

#Get Tenant ID

azure account show --json | jq -r '.[0].tenantId'

#Logout

azure logout

#Login with service principal and certificate

azure login --service-principal --tenant -u --certificate-file examplecert.pem --thumbprint

# The above command can be used in a script.  Be sure to include the certificate file in the same directory as your script (or add the path to it before the cert name).

[dboreham_FTNT]

Team,
One of my clients is using the supplied script and is running into issues.

Here is what they are seeing:

error: Error "Unexpected token <" occurred in deserializing the responseBody - "http://www.w3.org/TR/html4/strict.dtd">

Bad Request - Invalid URL

---

HTTP Error 400. The request URL is invalid.

Heres what they get when they ran a specific line manually.

root@ubuntuFWHAMON:~# azure network vnet subnet show -g "Film-Infra-VNET-DMZ-rg-NRG" -e Film-DMZ-UW-VNT-00 -n Film-FWADC-INT-SUB-00 --json | jq '.routeTable.id' | cut -d "/" -f9 | cut -d """ -f1

fortigateintfw2-APublicFacingSubnetRoutes

Please let me know if there are any suggestions.  I'm going to be doing addtional research on this, but wanted to add this to Fuse as well as it is time sensitive.

Regards,

-Doug

Doug Boreham | Principal Systems Engineer | FORTINET — US Enterprise East

Skype: doug.boreham

[dboreham_FTNT]

We resolved the issue.  We had to convert the script file to unix using dos2unix tool, as it has some extra characters in the script, which were causing the failures.

Regards,

-Doug

Doug Boreham | Principal Systems Engineer | FORTINET — US Enterprise East

Skype: doug.boreham

[dboreham_FTNT]

Some additional questions from our customer:

What are the requirements of the Azure network configuration to have this script to be successful?  Do Fortigate VMs, VNETS, UDR, Load Balancer, and Storage Accounts all need to be on the same resource group? Currently  They are having errors since the UDRs are on separate resource group.  Can they just move the UDR to same resource group as the VNETs.?

Here are the errors:

A route table with name "fortigateintfw2-AProtectedSubnetRoutes" not found in the resource group "Film-Infra-VNET-DMZ-rg-NRG"

Regards,

-Doug

Doug Boreham | Principal Systems Engineer | FORTINET — US Enterprise East

Skype: doug.boreham

[MartTwom]

Hi Doug,

Yes, the sample script provided does assume that all resources are in the same resource group. However, it is meant to be a sample and can certainly be modified to work with an existing environment. I can help with that if needed. It is important for customers to understand that the script is provided as an example of how Azure networks can be modified in an automated fashion. It's not a Fortinet product and comes with no official support.

Regards,
Martin



On Thu, 2016-08-25 at 15:26 -0700, Doug Boreham, NSE6 via cloudsecurity.pub wrote:

Some additional questions from our customer:

What are the requirements of the Azure network configuration to have this script to be successful? Do Fortigate VMs, VNETS, UDR, Load Balancer, and Storage Accounts all need to be on the same resource group? Currently They are having errors since the UDRs are on separate resource group. Can they just move the UDR to same resource group as the VNETs.?

Here are the errors:

A route table with name "fortigateintfw2-AProtectedSubnetRoutes" not found in the resource group "Film-Infra-VNET-DMZ-rg-NRG"

Regards,
-Doug

Doug Boreham | Principal Systems Engineer | FORTINET — US Enterprise East
Skype: doug.boreham

-----End Original Message-----

--
Martin Twombly
Cloud Architect

M: +1 909.257.8234
899 Kifer Road |
Sunnyvale, CA 94086
www.fortinet.com