External IP for VIP changes but no routing

[Eldrich]

Hi,

We are replacing a FG-80E with an FG-81F (7.2.8) and hit the following issue when changing the external IP address for our VIPs from test to live that the live DMZ machines are not contactable from the internet.

We have setup the FG-81F with a temp external IP address A.A.A.100 and two VIPS A.A.A.101 and A.A.A.102, these are both mapped to 10.10.10.13 and 10.10.10.14 and a test Linux machine is sitting there and is quite happy, it can be contacted from the outside and ping to the outside world. So it seems that the firewall policies are happy.

When we replace the IP addresses with the actual live ones A.A.A.50, and VIPS with A.A.A.51 and A.A.A.52 then our live Linux servers (10.10.10.13 and 10.10.10.14) can not be contacted externally and they can not ping out. They can be contacted from the LAN->DMZ though, if the machines on the LAN contact them via their external IP addressed they can, however external machines can not contact them.

The only changes we make from test to live is to change the FG-81F IP address and the 2 vip addresses (and to unplug the old GF-80 from the fiber). If we put the old FG-80E back in then the live Linux machines are available again. If we change the FG-81F's ip and vips back to the test ones and put the test LInux machine behind it then it works perfectly again. 

We are thinking this is some kind of ARP issue as we have a 2nd FG-81F with essentially the same setup and it works fine, we have matched the firewall policies between the two machines and they all match.

Is this a thing that if you change the vip's then you have to do something with the arp as well?

Thanks in advance

[ozkanaltas]

Hello @Eldrich ,

Firstly, do we need to make sure related traffic comes to FortiGate or not?

You can check this with the sniffer command. After running this command, you need to try to access your service

diagnose sniffer packet any 'host x.x.x.x' 4 a (x.x.x.x your public IP)

If you see a connection request after running this command. You can run these commands for debugging purposes. After running these commands Fortigate gives some output to you. Can you share those outputs with us? 

diagnose debug flow filter daddr x.x.x.x
diagnose debug flow trace start 100
diagnose debug enable

[AEK]

Hi Eldrich

Did you try to enable arp-reply for your VIPs?

config firewall vip
  edit vip1
    set arp-reply enable
  next
end

The default config for VIP's arp-reply depends on your firmware version. Check this article:

https://community.fortinet.com/t5/FortiGate/Technical-Tip-IP-pool-and-virtual-IP-behavior-changes-in...

[Eldrich]

Thanks for the replies, I was working on this all week and posted 4pm on Friday hoping to have some replies over the weekend to try on Monday. I will try these on Monday.

As a further info, for the live Linux machines in the DMZ, pinging out to a known external machine fails but a traceroute shows the path all the way to the external machine so I think the ping is going out and it is the return pong that is not being routed in.

We also do not reboot the FG-81F between changing IP and vips, I do wonder if we change the IP/vip and then rebooted if that would help. The difficulty is it works fine in the test environment but fails in the live and all we do to change between them is the IP/vips.

[Eldrich]

Quick update, we have not tried it live yet but using after changing the vips to the live ones, rebooting the FG-81F and testing it offline (laptop pretending to be internet instead of fiber modem) it seemed to work as expected, with the 'external' laptop pinging through to the test servers. I will schedule a time this week to try the replacement again and see what it like live. 

[Eldrich]

Just another follow up, we only have a 30 min window each week to make the change and do all tests before deciding whether use it or go back to the old machine. Art this point we still are failing, everything works fine except the internet to VIP servers. We're starting to think it is an ISP issue in that they are not recognising the change in the device handing the VIP as everything else seems to work fine, the diagnose command given earlier would confirm that for us and we'll run it next chance we have to try the switch.

[sw2090]

To me that looks like you try to access three wan ip while you only have one assigned by your isp. So external traffic to the other two cannot reach your FGT. You either have to have  more than one wan ip or you would have to use one wan ip and different ports on your vips.

[Eldrich · Accepted solution]

Just to follow up, after running the diagnostics suggested here it became obvious it was an arp issue with our ISP after contacting them and finding out the arp update was around 4 hrs they suggested:

set gratuitous-arp-interval 10

and it worked

After around 30 mins we then followed up with

set gratuitous-arp-interval 0

to stop the gratuitous-arp

so far so good

Thanks for all your replies