SSL VPN Saml authentication with Duo
We have been using SSL VPN with Azure AD without issue for a couple months now. I am currently doing a POC with Duo for mfa and am running into some problems with SSL VPN. The setup seems straight forward and Duo has some documentation on it, but I cannot get VPN to work. I get the Duo login screen and have to allow the MFA, then I just get a message saying "No Access."
So it seems like the Saml part is working it's just that the Fortigate is not giving my account access. The only difference in the config between this and Azure is that with Azure, we were using the group matching like so.
edit "VPNtest"
set member "azure"
config match
edit 1
set server-name "azure"
set group-name "object id from Azure AD"
With Duo, I simple want to allow all users so it is configured as:
edit "VPNtestduo"
set member "duo"
next
I tried running some debug commands for sslvpn, and saml, but I can't see why it isn't allowing access. Any ideas?