In GUI under Network->Interfaces, open and enter into Edit mode of the interface your public IP is configured, likely one of wan interfaces. In Administrative Access section, HTTPS and/or HTTP must be "checked". Just uncheck them. That would disable it. Make sure you do this while you're connected via VPN.
The change you would make is to just disable HTTPS/HTTP access to the wan interface with its public interface IP. It wouldn't affect any other interfaces including the internal interfaces you can reach over a VPN, and which I assume you've set up your admin access to with its interface (private) IP. You will keep using it without any additional change.
I'm assuming the VLAN interface is allowed for PING and you can ping it from internal device. Also assuming the VLAN subnet has been made reachable over VPN. Can you ping it over the VPN? If not you need to troubleshoot the reachability problem first.
The Fortinet Security Fabric brings together the concepts of convergence and consolidation to provide comprehensive cybersecurity protection for all users, devices, and applications and across all network edges.