Troubelshooting Tip: How to fix the uWSGI crashes issue due to old greenlet package version installed and integrated

Behavior:

FortiSOAR partition '/var/log' disk space consumption increases abnormally. The disk space usage is observed to be mainly consumed by the folder '/var/log/cyops/coredump'. Repeating coredump files generated for process 'uWSGI'.

Error:

Check the log file '/var/log/cyops/cyops-integrations/integrations.log' and locate the errors if identical to these messages:

...The work of process 1656432 is done. Seeya!
!!! uWSGI process 1656432 got Segmentation Fault !!!
*** backtrace of 1656432 ***
/opt/cyops/python_packages/third_party/bin/uwsgi(uwsgi_backtrace+0x2a) [0x46beca]
/opt/cyops/python_packages/third_party/bin/uwsgi(uwsgi_segfault+0x21) [0x46c281]
/usr/lib64/libc.so.6(+0x3e730) [0x7ff53223e730]
/opt/cyops/configs/integrations/packages/lib/python3.9/site-packages/greenlet/_greenlet.cpython-39-x86_64-linux-gnu.so(_ZN24ThreadState_DestroyNoGIL19DestroyQueueWithGILEPv+0x74) [0x7ff4f1e0b534]
/usr/lib64/libpython3.9.so.1.0(+0x96ccc) [0x7ff532696ccc]
/usr/lib64/libpython3.9.so.1.0(+0x69da6) [0x7ff532669da6]
/usr/lib64/libpython3.9.so.1.0(+0x14120b) [0x7ff53274120b]
/usr/lib64/libpython3.9.so.1.0(PyObject_VectorcallMethod+0x8b) [0x7ff53275e2bb]
/usr/lib64/libpython3.9.so.1.0(+0x211a4c) [0x7ff532811a4c]
/usr/lib64/libpython3.9.so.1.0(Py_FinalizeEx+0x37) [0x7ff532811777]
/opt/cyops/python_packages/third_party/bin/uwsgi(uwsgi_plugins_atexit+0x71) [0x468fb1]
/usr/lib64/libc.so.6(+0x40e2d) [0x7ff532240e2d]
/usr/lib64/libc.so.6(on_exit+0) [0x7ff532240f70]
/opt/cyops/python_packages/third_party/bin/uwsgi() [0x42309f]
/opt/cyops/python_packages/third_party/bin/uwsgi(simple_goodbye_cruel_world+0x57) [0x46b857]
/opt/cyops/python_packages/third_party/bin/uwsgi() [0x46b888]
/opt/cyops/python_packages/third_party/bin/uwsgi(uwsgi_close_request+0x53f) [0x423d0f]
/opt/cyops/python_packages/third_party/bin/uwsgi(simple_loop_run+0xc0) [0x467e80]
/opt/cyops/python_packages/third_party/bin/uwsgi(simple_loop+0x6f) [0x467f3f]
/opt/cyops/python_packages/third_party/bin/uwsgi(uwsgi_ignition+0x262) [0x46c582]
/opt/cyops/python_packages/third_party/bin/uwsgi(uwsgi_worker_run+0x23e) [0x470a6e]
/opt/cyops/python_packages/third_party/bin/uwsgi(uwsgi_run+0x41b) [0x470f9b]
/opt/cyops/python_packages/third_party/bin/uwsgi() [0x41ff0e]
/usr/lib64/libc.so.6(+0x295d0) [0x7ff5322295d0]
/usr/lib64/libc.so.6(__libc_start_main+0x80) [0x7ff532229680]
/opt/cyops/python_packages/third_party/bin/uwsgi(_start+0x25) [0x41ff35]
*** end of backtrace ***

Root cause:

The greenlet package installed and integrated into the FortiSOAR environment library is still on the old version:

# /opt/cyops-integrations/.env/bin/pip show greenlet
Name: greenlet
Version: 2.0.2
Summary: Lightweight in-process concurrent programming
Home-page: https://greenlet.readthedocs.io/
Author: Alexey Borzenkov
Author-email: snaury@gmail.com
License: MIT License
Location: /opt/cyops/configs/integrations/packages/lib/python3.9/site-packages
Requires:
Required-by: SQLAlchemy

# /opt/cyops-integrations/.env/bin/python3 -c "import greenlet; print(greenlet.__version__)"
2.0.2

Resolution:

Reinstall the greenlet package with the updated version.

1. Uninstall:

sudo env "PATH=$PATH" /opt/cyops-integrations/.env/bin/pip3 uninstall greenlet

2. Install greenlet-3.1.1 package:

sudo env "PATH=$PATH" /opt/cyops-integrations/.env/bin/pip3 install greenlet==3.1.1

3. Check installed package:

sudo env "PATH=$PATH" /usr/bin/pip3 show greenlet
/opt/cyops-integrations/.env/bin/python3 -c "import greenlet; print(greenlet.__version__)"

4. Restart service:

csadm services --restart

Note: 

Take a VM snapshot before reinstalling the package.