Management access from a specific outside site, I thought it was simple

I've done this countless times on non-Fortinet firewalls so the concepts are far from new for me.

I want to be able to access the management web page from the outside, from a specific IP address.

I do not want to limit in any way the access on other interfaces. Some of the subnets get changed and I don't want to use the permitted host in the management because this could result in the firewall not be accessible. I also need to use the same username outside as in.

Normally I would enable https management, and creat an ACL that permitted access to https, on the outside interface, from a specific subnet. And the implicit deny would take care of the rest.

But on the Fortigate when I enable the management access it lets in https from everywhere.

I tried creating a specific inbound policy limiting inbound https to the subnet, and a specific deny policy for https from everywhere (in sequence after the permit). But this seems to do nothing.