I cannot say for sure on the EMS/EDR front, but I guess that this will depend on some specific factor that the endpoint encryption supplies. I am assuming with endpoint encryption, you mean disk encryption.
Detecting this is difficult. Whether this is a specific registry value, file existence, daemon is difficult to say and will differ from the encryption implementation used.
I can imagine this being VERY difficult to detect if the disk is encrypted by its own native disk encryption (many SSDs offer this).
To an encrypted disk, you authenticate on device boot, not the OS.
- If you authenticate successfully, the OS is loaded.
- If you fail to authenticate the disk is unreadable, OS cannot be found.
- The disk encryption might be managed by BitLocker and then again you could detect it, then you authenticate to the bootloader, that passes this to the disks and keeps the "successful auth" info available for Windows authentication, such as having a single sign on.
If your clients are all the same type, this is maybe easier.
If they are different, you will have a hard time having them match policies.