Pods failing to unmount containerd 'tmpmount' paths when Lacework agent is deployed

[Kate_M]

Description	Pods are reporting errors when trying to unmount specific "tmpmount" paths. The error messages look like the following: unpacking linux/amd64 sha256:c33dd9a587560191b55d6c6831738f977f4452505199651756ba491a34b66dcb... INFO[0006] apply failure, attempting cleanup             error="failed to extract layer sha256:ccdbb80308cc5ef43b605ac28fac29c6a597f89f5a169bbedbb8dec29c987439: failed to unmount /group/app/containerd/tmpmounts/containerd-mount093668013: failed to unmount target /group/app/containerd/tmpmounts/containerd-mount093668013: device or resource busy: unknown" key="extract-906700158-sFgo sha256:ccdbb80308cc5ef43b605ac28fac29c6a597f89f5a169bbedbb8dec29c987439" ctr: failed to extract layer sha256:ccdbb80308cc5ef43b605ac28fac29c6a597f89f5a169bbedbb8dec29c987439: failed to unmount /group/app/containerd/tmpmounts/containerd-mount093668013: failed to unmount target /group/app/containerd/tmpmounts/containerd-mount093668013: device or resource busy: unknown The error only occurs when a Lacework agent is deployed and running.
Scope	Kubernetes clusters running Lacework agent pods
Solution	This issue is due to the Lacework agent accessing the mounted paths whilst scanning them as part of its File Integrity Monitoring ("FIM") capability. While the paths are still in use, containerd is unable to unmount them. As per this containerd Github issue: https://github.com/containerd/containerd/issues/5538 this problem can be triggered by any process that keeps the paths mounted for any amount of time; however it is noted that security scanning software is the most common use case.  Also as per the Github issue, the resolution is to configure the relevant software to ignore/bypass the temp mount paths. As the paths are mounted as part of containerd's internal operations and are transient copies of paths that are persistently mounted elsewhere on the host, we are safe to exclude these temporary duplicates from the FIM scanning activity. The specific action required in the case of the Lacework agent is to add the path(s) to the agent's "fileignore" FIM configuration property. An example of adding such a path to an existing fileignore config might look like this: Before: "fim": {     "fileignore": "/etc/mtab,/etc/mnttab,/etc/hosts.deny,/etc/mail/statistics" } After: "fim": {     "fileignore": "/etc/mtab,/etc/mnttab,/etc/hosts.deny,/etc/mail/statistics,/var/lib/containerd/tmpmounts/containerd-mount*" }