Troubleshooting Tip: FortiGate Azure showing 'virtual machine agent not ready' in console

[kcheng]

Description	This article describes how to resolve an issue with the Microsoft Azure Linux Agent (waagent) which manages Linux and FreeBSD provisioning and VM interaction with the Azure Fabric Controller. (For more information, refer to Azure Linux VM Agent overview).
Scope	FortiGate Azure.
Solution	FortiGate deployed in Azure is equipped with a Microsoft Azure Linux Agent during installation. Occasionally, the Microsoft Azure portal states that the virtual machine agent is not ready and requests troubleshooting the issue: When this issue occurs, it usually indicates a problem with connecting to the Azure Fabric Controller (168.63.129.16). Confirm this is the case with the following steps: To check the connectivity between the FortiGate virtual machine agent and Azure Fabric Controller, run the following command: diag deb app waagent -1  diag deb console timestamp enable diag deb en diag test app waagent 1 Note that the virtual agent is trying to connect to Azure Fabric Controller IP 168.63.129.16 and the connection did not manage to go through. Run the debug flow to check on the traffic flow: diag deb flow filter addr 168.63.129.16 diag deb console timestamp enable diag deb flow sh function-name en diag deb flow sh iprope en diag deb flow trace start 10000 diag deb en From the debug flow above, it is possible to see that the traffic towards the Azure Fabric Controller is being forwarded to port2 which is supposed to be meant for the internal network traffic. Check the routing table for 168.63.129.16 with the following command: get router info routing-table details 168.63.129.16 Note that there is a static route configured to route traffic destined to 168.63.129.16/32 to port2. The Azure Fabric Controller IP is considered an external IP from FortiGate's point of view.  As a result, the traffic should be exiting using port1 instead of port2. In the above scenario, removing the static route and re-initiating waagent connection will resolve the issue. Recheck the connection with the following commands, which should show that waagent is communicating successfully with Azure Fabric Controller: diag deb reset diag deb app waagent -1 diag deb console timestamp enable diag deb en diag test app waagent 1 Once the waagent is able to send the traffic towards the correct interface, the agent status and host details will appear in the Azure console respectively: The connection between the Microsoft Azure Agent and Azure Fabric controller will not impact FortiGate performance. There is no downtime required to perform the above remediation steps. There are cases in which the deployment uses an ELB (external load balancer) and ILB (internal load balancer)  As a result, 2 static routes are required: config router static     edit <static_route_index>         set dst 168.63.129.16 255.255.255.255         set gateway <gateway_IP>         set device "port2"     next     edit <static_route_index>         set dst 168.63.129.16 255.255.255.255         set gateway <gateway_IP>         set device "port1"     next end What will happen in this situation is that the waagent will use the internal port2 as mentioned before, and since we need the static route for ILB probing we cannot delete the route for port2. Sniffer packet. 2024-07-10 15:06:56.596646 port2 out 10.10.3.7.6332 -> 168.63.129.16.80: syn 3110387016  2024-07-10 15:06:56.596652 sriovslv1 out 10.10.3.7.6332 -> 168.63.129.16.80: syn 3110387016 2024-07-10 15:06:57.609800 port2 out 10.10.3.7.6332 -> 168.63.129.16.80: syn 3110387016 Waagent debug. 2024-07-10 15:06:56 waagent<15963> httpGet:560 URL: http://168.63.129.16/machine/?comp=goalstate <-- Communication with agent stopped. 2024-07-10 15:06:56 waagent<15963> waagent_curl_socket:45 connecting to 168.63.129.16:80:168.63.129.16 2024-07-10 15:07:06 waagent<15963> httpGet:584 res 28, code 0 2024-07-10 15:07:06 waagent<15963> httpGet:560 URL: http://168.63.129.16/machine/?comp=goalstate 2024-07-10 15:07:06 waagent<15963> waagent_curl_socket:45 connecting to 168.63.129.16:80:168.63.129.16  The solution for this case is to change the priorities of the static routes: config router static     edit <static_route_index>         set dst 168.63.129.16 255.255.255.255         set gateway <gateway_IP>         set device "port2"         set priority 2     next     edit <static_route_index>         set dst 168.63.129.16 255.255.255.255         set gateway <gateway_IP>         set device "port1"         set priority 1     next end This will allow the routing outgoing (ELB + waagent) through port1 and accepting egress traffic (from ILB to 168.63.129.16/32) from port2. For more information on deploying the FortiGate with ELB/ILB scenario, refer to the following document: Configuring FGSP session sync Note 1: If the message appears on the passive VM only in a deployment consisting of an HA cluster of Azure FortiGate VM running in an Active-Passive model, this is expected behavior. It occurs because, when running HA in Active-Passive mode, the passive device does not have the routing table. As a result, it will not be able to connect to the Azure Fabric Controller IP at 168.63.129.16/32. Note 2: If deploying with single Load balancer (ELB) traffic for Azure Fabric Controller IP at 168.63.129.16/32 router to an external connected port only as the traffic to Azure IP 168.63.129.16/32 should be flowing through port1 as it is external (Azure backbone) traffic. The agent is configured to communicate to Azure backbone IP to send host information and host monitoring purposes. Hence, the traffic should not be sent to the internal network (port2).