Technical Tip: Troubleshooting Address Flickering in FortiGate with VMware Connector

[lpizziniaco]

Description	This article describes how to troubleshoot address flickering in FortiGate caused by VMware connector issues. It covers key debugging steps on FortiGate and VMware technologies to understand eventual VM network configuration discrepancies.
Scope	FortiGate, Fabric Connector.
Solution	When address flickering occurs from a VMware connector on the FortiGate, logs such as the following can be observed (the logs below are shortened to increase readability): date=2024-05-22 time=01:17:27 type=event subtype=connector level=information logdesc="Dynamic address added" cfgobj="OBJECT_A" action=object-add addr="10.x.x.37" msg="connector object discovered in addr-obj OBJECT_A, 10.x.x.37" date=2024-05-22 time=01:16:28 type=event subtype=connector level=information logdesc="Dynamic address removed" cfgobj="OBJECT_A" action=object-remove addr="10.x.x.37" msg="connector object removed in addr-obj OBJECT_A, 10.x.x.37" Before opening a support ticket with Fortinet, it is important to verify a few elements and gather relevant data. The initial step can involve running some debug commands, such as: diagnose debug application vmwd -1 diagnose debug en Check for any error messages that appear. If there is any, and there is no easy solution, contact Fortinet support, attaching the collected debug information. If no error is detected, it may be useful to investigate on the VMware side using the Managed Object Browser (MOB) or VMware API. Since the connector makes API requests and updates addresses based on data retrieved from the server, frequent changes in the API response from the vSphere server could be causing the flickering. To investigate the issue in the vSphere Managed Object Browser, follow these steps, assuming the affected machine is identified by vm-ijk: Open the vSphere Managed Object Browser https://<vsphereIP>/mob. Navigate to ServiceContent -> Content (in the following x, y, and z are indexes assigned by vSphere). Go to rootFolder -> group-dx. Access childEntity -> datacenter-y. Proceed to vmFolder -> group-vz. Select childEntity -> vm-ijk. Go to guest -> guest. Refresh the page and if the 'net' property fluctuates between 'set' and 'unset' ('Unset' indicates that virtual NICs are missing). In this case, it is important to review the VM's network settings and consult VMware for troubleshooting. Another way to troubleshoot the issue is by using the vSphere API. The first request to perform is the log on the system. curl --silent --location --request POST 'https://<vsphereIP>/sdk' \ --header 'Content-Type: application/xml' \ --data-raw '<?xml version="1.0" encoding="UTF-8"?> <soapenv:Envelope xmlns:soapenc="http://schemas.xmlsoap.org/soap/encoding/" xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/" xmlns:xsd="http://www.w3.org/2001/XMLSchema" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">     <soapenv:Body>         <Login xmlns="urn:vim25">             <_this type="SessionManager">SessionManager</_this>             <userName>**user**</userName>             <password>**passwd**</password>         </Login>     </soapenv:Body> </soapenv:Envelope>' The session and the cookie key must be noted to get the information concerning the faulty VM (let's say vm-ijk) via the following request: curl --silent --location --request POST 'https://<vsphereIP>/sdk' \ --header 'Content-Type: application/atom+xml' \ --header 'Cookie: vmware_soap_session="**session**"' \ --data '<?xml version="1.0" encoding="UTF-8"?> <soapenv:Envelope xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/"     xmlns:vim25="urn:vim25">     <soapenv:Header>         <vcSessionCookie>**obtained key**</vcSessionCookie>     </soapenv:Header>     <soapenv:Body>         <RetrieveProperties xmlns="urn:vim25">             <_this type="PropertyCollector">propertyCollector</_this>             <specSet>                 <propSet>                     <type>VirtualMachine</type>                     <all>true</all> <!-- Set this to true to get all properties -->                 </propSet>                 <objectSet>                     <obj type="VirtualMachine">vm-ijk</obj>                     <skip>false</skip>                 </objectSet>             </specSet>         </RetrieveProperties>     </soapenv:Body> </soapenv:Envelope>' A lot of information will be shown but if performing the same request a few times the answer payload, in some cases, is missing <net> part like the following: <net>   <network>Vlan8</network>   <ipAddress>10.x.x.37</ipAddress>   <ipAddress>fe80::250:56ff:fea0:dc52</ipAddress>   <macAddress>00:50:56:aa:bb:cc</macAddress>   <connected>true</connected>   <deviceConfigId>4000</deviceConfigId> </net> That means that the issue is coming from VM network settings, and as in the previous example, the machine must be reviewed. In some cases, a VM reboot could be enough.   Once the VM issue is solved, the flickering logs should stop.