I have finally resolved it. Sharing what I found.
Found out Azure VMs has built-in NAT gateway that forces the internet outbound connection of VM. (Even if you change the actualy gateway IP inside the VM.) Thats why we get different IP when we run ipchicken or whatsmyip.
So there will be two things to fix it,
(1) additional routing within the Fortigate and
(2) create a table route from Azure Route Table and associate it to Vnet connected to Fortigate's internal subnet.
This link helped me, contains the needed IP routing:
After this, webserver could now get the Original IP of web visitors when NAT is disabled. Connection also does not break when disabling NAT. IPchicken on webserver is now the Fortigate-assigned Public IP.
Hope this help you also!