New Member

Question

Azure SDN fabric connector

Forum|Forum|4 years ago
January 20, 2022
8 replies
13588 views

Hello,

We've setup an FGVM cluster on our Azure tenant, based on Fortinet github template https://github.com/fortinet/azure-templates/tree/main/FortiGate/AvailabilityZones/Active-Passive-ELB-ILB-AZ.

I've originally setup the SDN connector to create firewall objects, and to have this done, following the documentation, i gave the "reader" permission on subscriptions to the two FGVM virtual machines on Azure. It worked for a while and i could create dynamic objects correctly.

Since some reboots and few days of exploitation, the connector has stopped working. Following that KB : https://docs.fortinet.com/document/fortigate-public-cloud/7.0.0/azure-administration-guide/985498/troubleshooting-azure-sdn-connector, here is the debug log i got:

azd sdn connector AzureSDN prepare to update
azd sdn connector AzureSDN start updater process 881
azd sdn connector AzureSDN start updating
azd updater process 881 is updating
azd updater process 881 is updating
curl DNS lookup failed: management.azure.com
azd api failed, url = https://management.azure.com/subscriptions?api-version=2018-06-01, rc = -1,
azd failed to list subscriptions
azd failed to get ip addr list
azd reap child pid: 881

"curl DNS lookup failed" : i don't understand, since a "ping management.azure.com" resolves correctly the address:

fgvm-appliance # exec ping management.azure.com
PING arm-frontdoor-prod.trafficmanager.net (40.79.131.240): 56 data bytes

The two DNS servers setup on the FGVM are reachable...

Here is the SDN connector configuration (default from github template):

config system sdn-connector
edit "AzureSDN"
set type azure
set ha-status enable
set update-interval 30
next

On trafic-side, if i try to traceroute that load-balancer IP 40.79.131.240... (i know this is one of the multiple IPs, but it's representative). The packet goes out by the WAN interface, from local. I can't trace once after that, it goes on Azure external load balancer and internet.

#execute traceroute 40.7.131.240
id=20085 trace_id=1 func=print_pkt_detail line=5783 msg="vd-root:0 received a packet(proto=1, [redacted:IP of WAN interface]:33727->40.79.131.240:2048) from local. type=8, code=0, id=33727, seq=1."
id=20085 trace_id=1 func=init_ip_session_common line=5955 msg="allocate a new session-000053df"
traceroute to 40.79.131.240 (40.79.131.240), 32 hops max, 3 probe packets per hop, 84 byte packets
1 *id=20085 trace_id=2 func=print_pkt_detail line=5783 msg="vd-root:0 received a packet(proto=1, [redacted:IP of WAN interface]:33727->40.79.131.240:2048) from local. type=8, code=0, id=33727, seq=2."

The default route is the WAN interface of the FGVM (port1), it's the default from the github template.

config router static
edit 1
set gateway [redacted: external load-balancer IP]
set device "port1"
next

Any ideas ?

infrasigrpAuthor

New Member

Hello everyone,

Some updates about our issue,

I noticed that when i setup the Fortiguard DNS, it work again.

The system DNS is configured with a source IP & interface, like that i can create the appropriates rules between the FGT and our DNS servers. It seems to work correctly, since i can ping everything from the Fortigate and see it resolved correctly.

But, the behavior from the SDN connector seems to be different : i tried to capture all the trafic between the FGT DNS source ip & my DNS server, i can see the packets containing the requests from my Azure servers, but nothing containing the "management.azure.com" or "graph.microsoft.com".

Do you know if there is an other DNS resolution mechanism used for SDN connectors, to call APIs ? I couldn't see any parameters regarding the network in the SDN configuration...

Thanks in advance,

Arnaud

infrasigrpAuthor

New Member

Hello everyone,

I confirmed that no DNS trafic come from the private IP i've set-up. The following is the source-IP configured from internal FGT services:

It seems that the source-ip setting is not taken in account...

This is the config:

Any ideas?

Hassan09

Staff

Hello,

It looks like you have disabled public IPs on MGMT interface. MGMT interfaces must also be able to access internet to interact with Azure Management API.

Try to run this command and check if the resolution is working:

#exec enter vsys_hamgm

#exe ping www.google.com

infrasigrpAuthor

New Member

Hello Hassan,

No, i haven't disabled public IPs on dedicated mgmt interfaces.

As well as:

Regards

Arnaud

DanielCastillo

New Member

I had the same issue:

Moreover, the FortiGate lost connectivity, all the interfaces (except the heartbeat) were brought down:

infrasigrpAuthor

New Member

Still an issue today. The SDN connector cannot contact Azure AD sometimes:

I'll definitely fallback on static / fqdn objects... shame.

DanielCastillo

New Member

I resolved this. You must use the dedicated management interface ALWAYS, so the fortigate can contact dns server and azure. The dedicated interface doesn't depend from the azure and SDN algorithm so it works.

kitzin

New Member

I had the same issue, but on our on-prem cluster (7.0.6).

I turned off "management interface reservation" in the HA configuration and the fabric connector started to work again.

As @DanielCastillo mentioned it seems to use the hidden management VDOM for communication so if you don't have any default route or can't route to Azure it won't work.

We're currently not using the individual management interfaces so I could just turn it off for now, but in air sealed out-of-band management it can be rough. Not sure if this should be considered a bug but we should at least be able to configure the source interface, IMO.

infrasigrpAuthor

New Member

Hello @kitzin @DanielCastillo

I can confirm that i have a default route on the FG via Azure, via an external loadbalancer. I need to conserve an individual access to each Fortigate, there is a public IP bound on corresponding NIC on Azure, that is the default configuration recommended by Fortinet in the Azure ARM templates https://github.com/fortinet/azure-templates/blob/main/FortiGate/Active-Passive-ELB-ILB/README.md. There is nothing particular about HA configuration for the SDN connector to work correctly... I don't know either if much people uses these templates...

New connector crash today,

azd api failed, url = https://management.azure.com/subscriptions/[redacted]/providers/Microsoft.Network/applicationGateways?api-version=2018-06-01, rc = 503 {"error":{"code":"ServerTimeout","message":"The request timed out."

then... the fortigate deletes cached dynamic addresses... ! causing network interruptions...

infrasigrpAuthor

New Member

About the SDN configuration, i just saw that there is two interesting settings "nic" & "route-table" which i cannot configure with a "set", some infos about that? :)

freddelm

Explorer

Any one find a resolution to this issue. Still experiencing this for issue on 7.0.12 on Azure.

tebby

New Member

I have been having this issue with one of our customers. No errors when running

diag debug application azd -1
diag debug enable
diag test application azd 4

or

diag debug application azd -1
diag debug enable
diag test application azd 99

However, the FGT was not able to pull in the resources related to newly created public IP addresses or fail over the IPs between FGTs.

The fix was to add the subscription ID and resource group name in the below section.

config system sdn-connector

edit "AzureSDN"

set type azure

set ha-status enable

set subscription-id "xxx"

set resource-group "piz-test"

I haven't required these 2 previously, but as soon as i added them and ran

diag debug application azd -1
diag debug enable
diag test application azd 99

the FGT pulled in the new objects, and when performaing a manual failover the public IPs moved from the primary to the secondary (now active) firewall.

I learned about the additinal commands from the following link

https://community.fortinet.com/t5/FortiGate-Cloud/Technical-Tip-A-first-steps-troubleshooting-guide-for-Azure/ta-p/239823

Sign up

Login with SSO

Login to the community

Login with SSO

Scanning file for viruses.

This file cannot be downloaded