Technical Tip: Avoiding upgrade failures and broken agents when using bulk provisioning tools

[Anthony_E]

Description	This article describes how to avoid upgrade failures and broken agents when using bulk provisioning tools.
Scope	FortiDLP.
Solution	Several users have observed problems with installing the Windows Reveal Agent using bulk provisioning tools. Symptoms: Before a reboot, an agent may continue reporting in activity events, but not process actions. Agents stop reporting any events or heartbeat messages, even if the machine is running. The Reveal Agent can no longer be removed from Add/Remove programs. Attempting to install a new version of the Reveal agent 'over the top' is unsuccessful. Affected users: Bulk software provisioning tools, including (but not limited to): Group Policy Application Management (GPO). System Center Configuration Manager (SCCM). Batch processes run as SYSTEM. Any upgrade or downgrade pushed via these tools. Any version of the Windows Reveal agent that does not include the MSIRMSHUTDOWN. Up to and including 4.0.0. Unaffected users: Agents installed manually (or via msiexec) as a standard admin user. Auto-upgrades pushed from the Fortinet Reveal platform, assuming it does not then conflict with a bulk provisioning tool and lead to a subsequent downgrade. Upgrades to agent version 4.0.1 onwards. Root Cause: The problem is caused by the fact that the Agent process has to run as the local SYSTEM user and that the majority of bulk provisioning tools work by triggering the Windows Installer (msiexec.exe) also running as the SYSTEM user. The agent is installed as normal and runs as a SYSTEM with a supervisor process (winsuper.exe) and several child processes (agent.exe). When an upgrade is triggered as SYSTEM, the installer immediately terminates the winsuper.exe process and service but does not exit the agent.exe cleanly. The agent.exe process still has active connections to the agent kernel drivers. The installer attempts to stop the drivers, which refuse as they are still active. After a 30-second timeout, the installer attempts to roll back the upgrade, but cannot stop the running driver(s). The installer fails with a fatal error (1603). The existing agent.exe process will keep running and report events back to the Fortinet Reveal platform until the system is restarted, heartbeats however will be terminated, and agent actions may no longer be processed. At this point, the drivers will have been deleted and not re-instated by the attempted rollback, meaning the agent will not start. Note: Those users installing or upgrading the MSI package manually, or invoking msiexec as a standard Administrator user are unaffected, as in this case the installer cleanly terminates the agent process and drivers.  Resolution: The only way to repair a system once it is in this state is to perform a hard purge of all Reveal application and driver binaries, as well as clear any associated registry entries. Fortinet Support can provide either Powershell or Command Prompt batch scripts that can perform these actions. Once the agent has been fully purged, it should be possible to push a new agent installer and automatically pick up the existing enrollment configuration. It is important to ensure that any upgrade pushed via a SYSTEM user includes the MSIRMSHUTDOWN=2 flag to msiexec. This ensures that the Windows Restart Manager cleanly terminates the agent process before upgrading. This has been included as standard in all versions of the auto-update process and is included in agent 4.0.1 or later MSI packages. If using SCCM as a provisioning tool, ensure that the instructions in this article have been followed correctly, paying particular attention to the setting on setting the deployment type to use the upgrade code to prevent SCCM from trying to downgrade after an agent auto-update has taken place.