Whenever we attempted to apply the HA configuration to either firewall:
config system ha set group-name "AzureHA" set mode a-p set hbdev "port4" 255 set session-pickup enable set session-pickup-connectionless enable set ha-mgmt-status enable config ha-mgmt-interfaces edit 1 set interface "port1" set gateway x.x.x.x next end set override disable set priority 255 set unicast-hb enable set unicast-hb-peerip x.x.x.x end
We would lose all connectivity to all interfaces the the firewall and from the console, we saw the following error:
[hagwnet_m_vd_policy_set:5047[ pid-234 ERROR args mlx4_en: sriovslv2: Failed changing HW MAC address mlx4_en: sriovslv1: Failed changing HW MAC address mlx4_en: sriovslv0: Failed changing HW MAC address mlx4_en: sriovslv3: Failed changing HW MAC address mlx4_en: sriovslv0: Failed changing HW MAC address port1: can not set mac address(22). mlx4_en: sriovslv2: Failed changing HW MAC address port2: can not set mac address(22). mlx4_en: sriovslv1: Failed changing HW MAC address port3: can not set mac address(22). mlx4_en: sriovslv3: Failed changing HW MAC address port4: can not set mac address(22).
To get the firewall back communicating again, we had to place it back into standalone mode from the console:
config system ha
set mode standalone
And sometimes, even that was not enough and we would have to reboot the firewall as well.
The error 'sriovslv' relates to SR-IOV (accelerated networking) and the virtual interfaces this feature creates.
Originally I thought we had an issue where by our FortiGate's physical interfaces were no tallying up with our virtual ones:
Running the hidden command 'fnsysctl ifconfig' shows all physical and virtual interface MAC addresses. You see that the virtual interfaces use the same MAC as the physical interface they are associated with. But in our case they matched as follows:
Port 1 = sriovslv1
Port 2 = sriovslv2
Port 3 = sriovslv3
Port 4 = sriovslv0
i.e. you'd have expected Port 1 = sriovslv0, Port 2= sriovslv1, etc...
At the same time we discovered this, TAC emailed to say they cannot replicate the same problem in their Azure environment but the only thing they did notice was our interfaces deployment was not the same as the Azure guide. We'd deployed the interfaces in the order:
Private Public HA-Sync Mgmt
But the documentation suggests:
Mgmt Public Private HA-Sync
We shut the Firewall VM down, re-arrange the interfaces and booted it back up. The firewall the accepted the HA config without an issue (or error). Therefore the order of the interface deployment must match the Fortinet documentation if you are looking at a HA deployment.
Running the commands fnsysctl ifconfig now still show some of the interfaces not matching what you'd expect their virtual interface numbers to be:
Port 1 = sriovslv1
Port 2 = sriovslv3
Port 3 = sriovslv0
Port 4 = sriovslv2
But it works so this is where my original logic was wrong...but got us on the right track about interface orders being wrong :)
Hope this help anyone who encounters the same issue. I could not find any information about it on the internet previously.
The Fortinet Security Fabric brings together the concepts of convergence and consolidation to provide comprehensive cybersecurity protection for all users, devices, and applications and across all network edges.