The CLI commands to create an FQDN-based VIP look like this:
config firewall vip
set type fqdn
set extaddr <fqdn-type-address-object>
set mapped-addr <fqdn-type-address-object>
set extintf <external-interface>
CLI documentation reference is available here. (6.2 link; the 6.0 document is unfortunately incorrect)
Note that for FQDN-type VIPs, the mapped destination is always mandatory to be an FQDN object, whereas the external address is optional (can be FQDN (set extaddr) or IP (set extip)). In other words, if you only need the external address to be an FQDN, you will need to set the internal one as FQDN as well.
Lastly, if you would like to see this in the GUI, the option was added in 6.4.2 (reference).
addendum: It may be worth pointing out, in case it is not clear, that these FQDNs' sole purpose is to provide dynamic updates to what would otherwise be the static-IP extip and mappedip attributes. The FortiGate periodically queries the DNS server for these FQDNs and uses the resulting IPs to internally update the extip/mappedip attributes of the VIP. This is to say, these FQDNs do not provide any sort of domain-based reverse-proxy functionality, nor any other "magic".
[ corrections always welcome ]