Mixed Local and SAML users with IPSEC Dialup

Hello all, in the context of migration from SSL-VPN I am facing the following problem.

Many of our client deployments include a mix of SAML users (dial up for company users) and locally defined users (contractors for instance)

In SSL-VPN it was as simple as enabling or disabling the SSO checkbox in Forticlient. If SSO was enabled, the connection flow would go through SAML authentication, and if disabled it would connect directly using local users.

I tried this with IPSEC dial up configuration, and unchecking SSO in Forticlient has no effect, it will always ask for SAML authentication.

Anyone tried this and got success? Any workaround?