Fiber ISP connection VLan

Hello , Everyone, My isp Deliver me a fiber connection over ONT, I've a Private router when I configure IPS and Vlan

Example: 

VLAN: 20 • IP Adress: 81.240.191.182 • WAN/Subnet Mask: 255.255.255.252/30 • Gateway: 81.240.191.181

I would like to know if it's possible remove my router and connect directly to my Fortigate VM64-HV or if I need to keep the router.