Azure Fortigate HA error - mlx4_en: sriovslv0: Failed changing HW MAC address

[JimmyHardman]

I am adding this to the forum for an issue we saw and eventually resolved with input from TAC.

We deployed two new Azure Foritgates (v7.0.6) roughly following the guide; https://github.com/fortinet/fortigate-terraform-deploy/tree/main/azure/7.0/ha-port1-mgmt-crosszone

Whenever we attempted to apply the HA configuration to either firewall:

config system ha
set group-name "AzureHA"
set mode a-p
set hbdev "port4" 255
set session-pickup enable
set session-pickup-connectionless enable
set ha-mgmt-status enable
config ha-mgmt-interfaces
edit 1
set interface "port1"
set gateway x.x.x.x
next
end
set override disable
set priority 255
set unicast-hb enable
set unicast-hb-peerip x.x.x.x
end

We would lose all connectivity to all interfaces the the firewall and from the console, we saw the following error:

[hagwnet_m_vd_policy_set:5047[ pid-234 ERROR args
mlx4_en: sriovslv2: Failed changing HW MAC address
mlx4_en: sriovslv1: Failed changing HW MAC address
mlx4_en: sriovslv0: Failed changing HW MAC address
mlx4_en: sriovslv3: Failed changing HW MAC address
mlx4_en: sriovslv0: Failed changing HW MAC address
port1: can not set mac address(22).
mlx4_en: sriovslv2: Failed changing HW MAC address
port2: can not set mac address(22).
mlx4_en: sriovslv1: Failed changing HW MAC address
port3: can not set mac address(22).
mlx4_en: sriovslv3: Failed changing HW MAC address
port4: can not set mac address(22).

To get the firewall back communicating again, we had to place it back into standalone mode from the console:

config system ha

set mode standalone

end

And sometimes, even that was not enough and we would have to reboot the firewall as well.

The error 'sriovslv' relates to SR-IOV (accelerated networking) and the virtual interfaces this feature creates. 

https://docs.fortinet.com/document/fortigate/6.2.0/new-features/651644/sr-iov-support-6-2-1

Originally I thought we had an issue where by our FortiGate's physical interfaces were no tallying up with our virtual ones:

Running the hidden command 'fnsysctl ifconfig' shows all physical and virtual interface MAC addresses.  You see that the virtual interfaces use the same MAC as the physical interface they are associated with. But in our case they matched as follows:

Port 1 = sriovslv1

Port 2 = sriovslv2

Port 3 = sriovslv3

Port 4 = sriovslv0 

i.e. you'd have expected Port 1 = sriovslv0, Port 2= sriovslv1, etc...

At the same time we discovered this, TAC emailed to say they cannot replicate the same problem in their Azure environment but the only thing they did notice was our interfaces deployment was not the same as the Azure guide.  We'd deployed the interfaces in the order:

Private
Public
HA-Sync
Mgmt

But the documentation suggests:

Mgmt
Public
Private
HA-Sync

We shut the Firewall VM down, re-arrange the interfaces and booted it back up.  The firewall the accepted the HA config without an issue (or error). Therefore the order of the interface deployment must match the Fortinet documentation if you are looking at a HA deployment.

Running the commands fnsysctl ifconfig now still show some of the interfaces not matching what you'd expect their virtual interface numbers to be:

Port 1 = sriovslv1

Port 2 = sriovslv3

Port 3 = sriovslv0

Port 4 = sriovslv2

But it works so this is where my original logic was wrong...but got us on the right track about interface orders being wrong :)

Hope this help anyone who encounters the same issue.  I could not find any information about it on the internet previously.

[Anonymous]

Hi 

Thanks for sharing this solution with our community

Looks very granular indeed !