Troubleshooting Tip: FortiWeb backup generation is slow or it is not working when Machine Learning is enabled

[AACastillo]

Description	This article describes when a FortiWeb backup generation takes too much time (or the backup process does not finish) because the Machine Learning information is included and how to solve this issue.
Scope	FortiWeb, FortiWeb-VM.
Solution	In a FortiWeb that works with Machine Learning settings enabled and it is saving a lot of information related to Machine Learning, sometimes FortiWeb could take too much time to generate a backup file (or the backup process does not finish). This happens when a FortiWeb backup file is saved using the GUI location System -> Maintenance -> Backup & Restore, in Backup & Restore -> Backup/Restore choose 'Back up entire configuration' with Include Machine Learning Data enabled and finally select Backup. The backup saving process begins but takes many minutes (or hours) or it never ends. In FortiWeb GUI it also happens in the location System -> Maintenance -> Backup & Restore, in Local Backup & Restore tab choosing Full Config, enabling Include Machine Learning Data, selecting Backup, giving a name to the backup file, and selecting OK. In the same way, the problem is present in FortiWeb CLI using the command 'exec backup full-config-with-ML-data' and finishes showing an error. On the other hand, if other options to save the backup are selected, like backup only CLI configuration or WAF configuration (in GUI or CLI), the FortiWeb backup file is generated correctly and fast (this can work as a workaround). To solve it,  verify the Machine Learning database distribution with the following command: diag sys redis status This command shows the used memory of different Machine Learning settings: Anomaly detection, Bot detection, and API detection: FortiWeb # diag sys redis status ml-anomaly-detection   :  maxmemory:430874344     used_memory:409916512  <------- Initial used memory. ml-bot-detection       :  maxmemory:22123504      used_memory:1151984 ml-api-protection      :  maxmemory:32790224      used_memory:11818704 user-tracking          :  maxmemory:6309065728    used_memory:872248 site-publish           :  maxmemory:3154532352    used_memory:872120 web-cache              :  maxmemory:12618132480   used_memory:970528 client-management      :  maxmemory:3154532352    used_memory:252616560 fctems                 :  maxmemory:3154532352    used_memory:893128 Clean and rebuild the database for Machine Learning with this command (Important: After the command finishes, it will reboot the FortiWeb). exec redis rebuild FortiWeb # exec redis rebuild This operation will clean database for ML and Client Management, and it will reboot the system! Do you want to continue? (y/n)y rebuilding the redis database...... When the command finishes the redistribution, check the Machine Learning used memory: FortiWeb # diag sys redis status ml-anomaly-detection   : maxmemory:18927198208     used_memory:1327648  <---- Used memory after ML Database redistribution. ml-bot-detection       : maxmemory:3154532352      used_memory:1151872 ml-api-protection      : maxmemory:3154532352      used_memory:1166888 user-tracking          : maxmemory:6309065728      used_memory:872216 site-publish           : maxmemory:3154532352      used_memory:872120 web-cache              : maxmemory:12618132480     used_memory:933952 client-management      : maxmemory:3154532352      used_memory:935048 fctems                 : maxmemory:3154532352      used_memory:893128 Check the FortiWeb backup generation with Include Machine Learning Data enabled: Related document: Backup & restore