Redhat Enterprise Linux 8 introduces "file access policy daemon"; this is an additional layer of file access control on top of existing UNIX permisions, ACLs, SELinux etc. https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/8/html/security_hardening/as...
In cases where the agent is failing to start, or upgrade, and the datacollector.log is showing permission errors like so:
time="2022-11-24T13:54:41.896Z" level=warning msg="Failed to start child fork/exec /var/lib/lacework/datacollector: operation not permitted" caller="controller.go:746" pid=3435
time="2022-11-24T13:54:44.898Z" level=warning msg="Failed to start child fork/exec /var/lib/lacework/datacollector: operation not permitted" caller="controller.go:746" pid=3435
time="2022-11-24T13:54:49.900Z" level=warning msg="Failed to start child fork/exec /var/lib/lacework/datacollector: operation not permitted" caller="controller.go:746" pid=3435
time="2022-11-24T13:54:50.901Z" level=info msg="Starting launch for : [/var/lib/lacework/lkg/datacollector]" caller="controller.go:766" pid=3435
if the host OS is a version of RHEL8, the fapolicy subsystem should be considered as a potential causative factor.
A quick check to see if fapolicyd is enabled and running is the systemctl status command:
# systemctl status fapolicyd
fapolicyd.service - File Access Policy Daemon
Loaded: loaded (/usr/lib/systemd/system/fapolicyd.service; enabled; vendor preset: disabled) Active: active (running) since Fri 2022-04-01 12:10:00 UTC; 17min ago Process: 753 ExecStart=/usr/sbin/fapolicyd (code=exited, status=0/SUCCESS) Main PID: 758 (fapolicyd) Tasks: 4 (limit: 48124) Memory: 154.5M CGroup: /system.slice/fapolicyd.service └─758 /usr/sbin/fapolicyd
Apr 01 12:10:01 ip-172-31-17-123.ec2.internal fapolicyd[758]: No SHA256 for /lib/systemd/system/datacollector.service
Apr 01 12:10:01 ip-172-31-17-123.ec2.internal fapolicyd[758]: No SHA256 for /var/lib/lacework/5.0.0.5826/datacollector
Apr 01 12:10:01 ip-172-31-17-123.ec2.internal fapolicyd[758]: No SHA256 for /var/lib/lacework/README
Apr 01 12:10:02 ip-172-31-17-123.ec2.internal fapolicyd[758]: Checking database
Apr 01 12:10:02 ip-172-31-17-123.ec2.internal fapolicyd[758]: Importing data from rpmdb backend
Apr 01 12:10:02 ip-172-31-17-123.ec2.internal fapolicyd[758]: Importing data from file backend
Apr 01 12:10:02 ip-172-31-17-123.ec2.internal fapolicyd[758]: Entries in DB: 21175
Apr 01 12:10:02 ip-172-31-17-123.ec2.internal fapolicyd[758]: Loaded from all backends(without duplicates): 21175
Apr 01 12:10:02 ip-172-31-17-123.ec2.internal fapolicyd[758]: Database checks OK
Apr 01 12:10:02 ip-172-31-17-123.ec2.internal fapolicyd[758]: Starting to listen for events
fapolicyd uses the concepts of user rules and "trusted files" in combination to create a very granular RBAC layer.
Note in the above output that it imports data from rpmdb, and a "file backend". By default, all file paths that are installed as part of an rpm package are implicitly trusted; as new rpms are installed, fapolicyd has an rpmdb-plugin to capture these activities and consume the relevant file paths.
Additional paths can be manually added by the administrator, with varying access rules - these are added to the "file backend" referenced above. Together with the rpmdb file lists, these make up the definitive "trust database" of files that are trusted.
To see the current rules in place, run the following:
sudo fapolicyd-cli --list
This lists out all the filesystem rules. A typical output may look like this:
-> %languages=application/x-bytecode.ocaml,application/x-bytecode.python,application/java-archive,text/x-java,application/x-java-applet,text/javascript,text/x-awk,text/x-gawk,text/x-lisp,application/x-elc,text/x-lua,text/x-m4,text/x-perl,text/x-php,text/x-python,text/x-R,text/x-ruby,text/x-script.guile,text/x-tcl,text/x-luatex,text/x-systemtap
1. allow perm=any uid=0 : dir=/var/tmp/
2. allow perm=any uid=0 trust=1 : all
3. deny_audit perm=any pattern=ld_so : all
4. allow perm=open exe=/usr/bin/rpm : all
5. allow perm=open exe=/usr/libexec/platform-python3.6 comm=dnf : all
6. allow perm=open all : ftype=application/x-sharedlib trust=1
7. deny_audit perm=open all : ftype=application/x-sharedlib
8. allow perm=execute all : trust=1
9. allow perm=any uid=0 : dir=/tmp/ansible
10. allow perm=any uid=0 : dir=/root/.ansible/tmp/
11. allow perm=open all : ftype=%languages trust=1
12. deny_audit perm=any all : ftype=%languages
13. allow perm=any all : ftype=text/x-shellscript
14. deny_audit perm=execute all : all
15. allow perm=open all : all
It's fairly self-explanatory; for example rule 1 allows uid 0 (root user) full permission to /var/tmp. Rule 2 gives root user full permissions to any file that has been marked as trusted (trust=1).
To see the current list of trusted files, run:
# fapolicyd-cli -D
On it's own this can be a very big list, so more useful would be to filter for paths of interest:
# fapolicyd-cli -D|grep -e lacework -e datacollector
rpmdb /etc/init.d/datacollector 2999 e2d1db7cd36584e41b9df0c05ea3566e
rpmdb /etc/init/datacollector.conf 609 ddcab5fc532e0cf846e53f7ffef44c6a
rpmdb /lib/systemd/system/datacollector.service 498 dfd3bbd0fa02cce291df5228bc1402c7
rpmdb /var/lib/lacework/4.1.62/datacollector 23586608 ceba6074e9d4ad416c760d2b51e4c6ce
rpmdb /var/lib/lacework/README 18 b3d7330b226ab021ea40a9f0d4e3e728
rpmdb /var/lib/lacework/config/config.json.example 1473 0734ae869377c2a3cf8e5f4279fb37a0
(Note: the above list might seem rather sparse; however fapolicyd is, as per its name, a file access layer. If we list the contents of the lacework rpm, the above six entries are in fact the only files included; all others in the list are directories or symlinks.
If the rules or trust database indicate a lack of permissions for files in /var/lib/lacework,or are referencing the wrong version (in particular the datacollector binary) this would be a strong indicator of the problem. Reasons for fapolicyd being incorrect can include the rpmdb plugin stopping, or the fifo pipe being deleted from /run/fapolicyd.
As a demonstration, if we start with these files being trusted:
# fapolicyd-cli -D|grep -e lacework -e datacollector
rpmdb /etc/init.d/datacollector 2999 e2d1db7cd36584e41b9df0c05ea3566e
rpmdb /etc/init/datacollector.conf 609 ddcab5fc532e0cf846e53f7ffef44c6a
rpmdb /lib/systemd/system/datacollector.service 498 dfd3bbd0fa02cce291df5228bc1402c7
rpmdb /var/lib/lacework/4.1.62/datacollector 23586608 ceba6074e9d4ad416c760d2b51e4c6ce
rpmdb /var/lib/lacework/README 18 b3d7330b226ab021ea40a9f0d4e3e728
rpmdb /var/lib/lacework/config/config.json.example 1473 0734ae869377c2a3cf8e5f4279fb37a0
then run the install.sh from a later version of datacollector, in normal operation fapolicyd detects this and updates accordingly:
# fapolicyd-cli -D|grep -e lacework -e datacollector
rpmdb /etc/init.d/datacollector 2999 e2d1db7cd36584e41b9df0c05ea3566e
rpmdb /etc/init/datacollector.conf 609 ddcab5fc532e0cf846e53f7ffef44c6a
rpmdb /lib/systemd/system/datacollector.service 498 dfd3bbd0fa02cce291df5228bc1402c7
rpmdb /var/lib/lacework/5.0.0.5826/datacollector 31910312 5130d628e8ce2a4d3b609f7c37fe39c5
rpmdb /var/lib/lacework/README 18 b3d7330b226ab021ea40a9f0d4e3e728
rpmdb /var/lib/lacework/config/config.json.example 1473 0734ae869377c2a3cf8e5f4279fb37a0
If, however, I delete the fifo file:
# cd /run/fapolicyd
[root@ip-172-31-17-123 fapolicyd]# ll
total 0
prw-rw----. 1 root fapolicyd 0 Apr 1 12:47 fapolicyd.fifo
[root@ip-172-31-17-123 fapolicyd]# rm fapolicyd.fifo
rm: remove fifo 'fapolicyd.fifo'? y
[root@ip-172-31-17-123 fapolicyd]# ll
total 0
and then repeat the exercise, the trusted files list remain unchanged post-upgrade.
[root@ip-172-31-17-123 5.0.0.5826]# fapolicyd-cli -D|grep -e lacework -e datacollector
rpmdb /etc/init.d/datacollector 2999 e2d1db7cd36584e41b9df0c05ea3566e
rpmdb /etc/init/datacollector.conf 609 ddcab5fc532e0cf846e53f7ffef44c6a
rpmdb /lib/systemd/system/datacollector.service 498 dfd3bbd0fa02cce291df5228bc1402c7
rpmdb /var/lib/lacework/4.1.62/datacollector 23586608 ceba6074e9d4ad416c760d2b51e4c6ce
rpmdb /var/lib/lacework/README 18 b3d7330b226ab021ea40a9f0d4e3e728
rpmdb /var/lib/lacework/config/config.json.example 1473 0734ae869377c2a3cf8e5f4279fb37a0
Workaround: A restart of fapolicyd in this case both reinstates the fifo file, AND updates itself from the rpmdb with the new files:
# systemctl restart fapolicyd
[root@ip-172-31-17-123 5.0.0.5826]# fapolicyd-cli -D|grep -e lacework -e datacollector
rpmdb /etc/init.d/datacollector 2999 e2d1db7cd36584e41b9df0c05ea3566e
rpmdb /etc/init/datacollector.conf 609 ddcab5fc532e0cf846e53f7ffef44c6a
rpmdb /lib/systemd/system/datacollector.service 498 dfd3bbd0fa02cce291df5228bc1402c7
rpmdb /var/lib/lacework/5.0.0.5826/datacollector 31910312 5130d628e8ce2a4d3b609f7c37fe39c5
rpmdb /var/lib/lacework/README 18 b3d7330b226ab021ea40a9f0d4e3e728
rpmdb /var/lib/lacework/config/config.json.example 1473 0734ae869377c2a3cf8e5f4279fb37a0
Manual trust addition
Ordinarily, it shouldn't be needed to add lacework file paths manually to the trust database, but the steps are documented here for reference.
This is a two-step process; first, adding the paths to the database, then informing fapolicyd to re-scan it's database for changes.
sudo fapolicyd-cli add /var/lib/lacework/
sudo fapolicyd-cli update
Note that while we supplied a directory path, fapolicy took that directory and scanned it for file paths that are then added individually to the trust db. If files are added to /var/lib/lacework subsequent to this update, they will not automatically be added.
|