Technical Tip: A first steps troubleshooting guide for Azure FortiGate

lpizziniaco · ‎12-19-2022

Description

This article contains an easy troubleshooting guide for users struggling with a HA FortiGate deployment on Microsoft Azure, including steps users are strongly advised to take before submitting a support ticket.

Scope

Azure FortiGate 6.4.2 and later.

Solution

Configuring a FortiGate on an Azure environment is not always an easy task. Some difficulties may be experienced during the configuration procedure, causing the FortiGate to not respond as expected. This guide aims to explain how to perform the very first troubleshooting steps for any issue encountered after deploying FortiGate. This guide assumes that no policies are applied at the Azure management level that deny any of the actions described below.

Subscription configuration

A helpful initial step before opening a support ticket is to check the subscription. Any issue with the subscription must be addressed to Microsoft and not to Fortinet.

It is important to activate all necessary features to allow all Virtual Machines to communicate: Microsoft.Network and Microsoft.ClassicNetwork must be registered in the resource provider. If they are not, errors like the following may appear:

"azd api failed",
"url = https"://management.azure.com/subscriptions/xyzt-ijkq/providers/Microsoft.Network/locations/westus/serviceTags?api-version=2020-05-01,
rc = 409,
,
{

"error":{

"code":"SubscriptionNotRegistered",

"message":"Subscription xyzt-ijkq is not registered with NRP.",

"details":[

]

}

FortiGate Azure SDN connector configuration

Once the configuration of subscription is complete, it's necessary to deploy an HA A-P cluster in which the handover is managed by the SDN connector. The procedure to follow is described in the official documentation linked below:

https://github.com/fortinet/azure-templates/tree/main/FortiGate/Active-Passive-SDN

Be aware that HA configuration and operation in cloud environments is very different from HA used with hardware appliances.

At this point, if an issue with the failover handling is experienced, it is important to verify the validity of the configuration of the SDN Connector as described in the documentation below:

https://github.com/fortinet/azure-templates/blob/main/FortiGate/Active-Passive-SDN/doc/config-failov...

The following is an example of properly configured active and passive FortiGate devices in Azure:

Host 1 SDN Configuration

Host 2 SDN Configuration

# config system sdn-connector

edit "AzureSDN"

set type azure

set ha-status enable

set subscription-id "xxx"

set resource-group "piz-test"

config nic

edit "piz-FGT-A-Nic1"

config ip

edit "ipconfig1"

set public-ip "piz-FGT-PIP"

config route-table

edit "piz-RouteTable-ProtectedSubnet"

config route

edit "toDefault"

set next-hop "10.1.9.4"

# config system sdn-connector

edit "AzureSDN"

set type azure

set ha-status enable

set subscription-id "xxx"

set resource-group "piz-test"

config nic

edit "piz-FGT-B-Nic1"

config ip

edit "ipconfig1"

set public-ip "piz-FGT-PIP"

config route-table

edit "piz-RouteTable

ProtectedSubnet"

config route

edit "toDefault"

set next-hop "10.1.9.5"

If Host 1 is the active FortiGate, the Azure Public IP address object piz-FGT-PIP is assigned to its Network Interface object piz-FGT-A-Nic1. This means that traffic from the internet to this public IP will be routed to Host 1’s first network interface.

During a failover, the (formerly) passive FortiGate takes control, becomes active, and issues API calls to Azure to shift the public IP address and update the internal user-defined routing to itself. This effectively switches the internal and external traffic to Host 2.

Notice that Host 2’s configuration dealing with the routing table is different on the primary and secondary devices. This is because their IP addresses are different and the route must point to the IP of the currently active device.

Additionally, the names of FortiGate’s interfaces for public IP address are different because the Network Interface objects assigned to each device have different names in Azure.

Updating the public IP address and gateway IP addresses of the routes takes a few seconds for Azure to complete.

Troubleshooting

Before opening a ticket, collect the FortiGate logs produced by the following commands:

# diag debug application azd -1
diag debug enable
diag test application azd 4

The last command is a test to verify the communication between the FortiGate and the Azure infrastructure. When it is successful, it generates a trace similar to the following:

azd stats: primary
azd sdn connector AzureSDN getting token
token size: 1287
token expire in: 3600 seconds
AzureSDN: resourcegroup: piz-lab, sub: xxx
Disable interface: port1
Disable interface: port2
get pubip piz-FGT-PIP in resource group piz-lab
found pub ip piz-FGT-PIP in resource group piz-lab
ipconfig id: /subscriptions/xxx/resourceGroups/piz-lab/providers/Microsoft.Network/networkInterfaces/piz-FGT-A-Nic1/ipConfigurations/ipconfig1
nic piz-FGT-A-Nic1 ipconfig ipconfig1 already has public ip piz-FGT-PIP from resource group piz-lab
query route table piz-RouteTable-ProtectedSubnet in resource group piz-lab of subscription a90dc88c-8dd5-4f94-a346-96eca9d90e0e
route table query, rc: 0
matching route:toDefault:toDefault
route toDefault already has nexthop 10.1.9.4
no route needs to be updated
Enable interface: port1
Enable interface: port2
refreshing public IP in interface config
getting instance metadata
NIC: 10.1.8.4, public IP: 20.16.234.101
NIC: 10.1.9.4
NIC: 10.1.10.4
NIC: 10.1.11.4, public IP: 20.16.231.123

The same command provides hints if something is going wrong due to a faulty configuration. For example, a configuration with incorrect non-managed identity credentials will cause output similar to the following to appear:

azd stats: primary
azd sdn connector AzureSDN getting token
401
get token failed
{
"error":"invalid_client",
"error_description":"AADSTS7000215: Invalid client secret provided. Ensure the secret being sent in the request is the client secret value, not the client secret ID, for a secret added to app xxx'.\r\nTrace ID: 5cb514b0-9bb5-478e-831e-78569ba74000\r\nCorrelation ID: 750e38e2-cca2-4d63-9b4c-e70ed855232a\r\nTimestamp: 2022-12-07 10:49:29Z",
"error_codes":[
7000215
],
"timestamp":"2022-12-07 10:49:29Z",
"trace_id":"5cb514b0-9bb5-478e-831e-78569ba74000",
"correlation_id":"750e38e2-cca2-4d63-9b4c-e70ed855232a",
"error_uri":"https://login.microsoftonline.com/error?code=7000215"
}
AzureSDN: resourcegroup: piz-lab, sub: a90dc88c-8dd5-4f94-a346-96eca9d90e0e
Disable interface: port1
Disable interface: port2
get pubip piz-FGT-PIP in resource group piz-lab
azd api failed, url = https://management.azure.com/subscriptions/xxx/resourceGroups/piz-lab/providers/Microsoft.Network/pu..., rc = 401
{
"error":{
"code":"AuthenticationFailed",
"message":"Authentication failed. The 'Authorization' header is missing."
}
}

In this example, something is going wrong with the authentication of the FortiGate in the Azure system and the root cause seems to be an invalid client secret provided. The fix in this case is simple to apply.

Before opening any ticket to Fortinet, it is important to identify (and if possible, fix) all errors reported by the API test.

A more extreme test to perform on a faulty FortiGate is the following:

# diag debug application azd -1
diag debug enable
diag test application azd 99

Test 99 restarts the azd daemon on the FortiGate. It will produce a very similar trace to the test 4, but the output will be more verbose since it will perform all checks relating to routing and interfaces.

In the case of subscription errors, it is advised to check the Azure configuration before reporting the issue. When a ticket is opened, the same trace must be attached to the ticket's body.

When a wrongly handled failover is experienced, the trace itself must be attached to the ticket. To get the failover trace, execute the following commands in the secondary and primary host of the cluster:

# diag debug application azd -1
diag debug enable

To simulate a failover (for 6.4.x and later), run execute ha failover set 1 on the primary CLI. When testing is finished, run execute ha failover unset 1.

The following is an example of a correct handover trace:

HA event
HA state: primary
azd sdn connector AzureSDN getting token
token size: 1287
token expire in: 3600 seconds
AzureSDN: resourcegroup: piz-lab, sub: xxx
Disable interface: port1
Disable interface: port2
get pubip piz-FGT-PIP in resource group piz-lab
found pub ip piz-FGT-PIP in resource group piz-lab
ipconfig id: /subscriptions/a90dc88c-8dd5-4f94-a346-96eca9d90e0e/resourceGroups/piz-lab/providers/Microsoft.Network/networkInterfaces/piz-FGT-B-Nic1/ipConfigurations/ipconfig1
removing pubip
query nic piz-FGT-B-Nic1
query nic piz-FGT-B-Nic1, rc: 0
remove public ip piz-FGT-PIP in ipconfig ipconfig1 of nic piz-FGT-B-Nic1
updating nic: piz-FGT-B-Nic1
updating nic: piz-FGT-B-Nic1, rc: 0
operation: "updating nic: piz-FGT-B-Nic1", status: Succeeded
adding pubip
query nic piz-FGT-A-Nic1
query nic piz-FGT-A-Nic1, rc: 0
add public ip piz-FGT-PIP in ipconfig ipconfig1 of nic piz-FGT-A-Nic1
updating nic: piz-FGT-A-Nic1
updating nic: piz-FGT-A-Nic1, rc: 0
operation: "updating nic: piz-FGT-A-Nic1", status: InProgress
operation: "updating nic: piz-FGT-A-Nic1", status: Succeeded
query route table piz-RouteTable-ProtectedSubnet in resource group piz-lab of subscription a90dc88c-8dd5-4f94-a346-96eca9d90e0e
route table query, rc: 0
matching route:toDefault:toDefault
set route toDefault nexthop 10.1.9.4
updating route table piz-RouteTable-ProtectedSubnet in resource group piz-lab of subscription a90dc88c-8dd5-4f94-a346-96eca9d90e0e
updating route table piz-RouteTable-ProtectedSubnet in resource group piz-lab of subscription a90dc88c-8dd5-4f94-a346-96eca9d90e0e, rc: 0
operation: "updating route table piz-RouteTable-ProtectedSubnet in resource group piz-lab of subscription a90dc88c-8dd5-4f94-a346-96eca9d90e0e", status: Succeeded
Enable interface: port1
Enable interface: port2
refreshing public IP in interface config
getting instance metadata
NIC: 10.1.8.4, public IP: 20.16.234.101
NIC: 10.1.9.4
NIC: 10.1.10.4
NIC: 10.1.11.4, public IP: 20.16.231.123
HA event
azd sdn connector AzureSDN prepare to update
azd sdn connector AzureSDN getting token
token size: 1287
token expire in: 3600 seconds
azd sdn connector AzureSDN start updater process 675
azd sdn connector AzureSDN start updating
azd sdn connector AzureSDN found 8 NICs from Azure
azd sdn connector AzureSDN get page 1 of vm list
azd vm list page size: 0
azd sdn connector AzureSDN found 0 VMs from Azure
HA state: primary
azd sdn connector AzureSDN getting token

Supported configurations

As a general rule, it is important to deploy the FortiGate while following the guidelines recommended in official Fortinet documentation:

https://docs.fortinet.com/document/fortigate-public-cloud/7.2.0/azure-administration-guide/128029/ab...

Fortinet cannot deny nor guarantee that any other unusual deployment can run correctly, but most variations are unlikely to be supported.