Proper way to setup multiple public ip addresses with multiple web servers

[landonious]

Hello. We have 3 public ip addresses and 2 web servers. I would like both to be behind the FortiGate but I want to make sure I set it up properly. From what I understand, I am not supposed to use both WAN interfaces and instead I am supposed to assign multiple ip addresses to one interface. If it matters, one of our ip addresses is on one subnet and the other two ip addresses are on a separate subnet. Ideally, the two webservers would use the single ip address and one of the other two. The only reason is that the dns records already point to these two and i don't want to change them if i don't have to. But if it's easier or better to use the two ip addresses that are on the same subnet, I can update everything as needed. I will still be using the same Fortigate and internet connection for all the devices on the network as well. What all do I need to do to set this up properly? Let's say the following:

Public IP address info:

Static Ip 123.45.25.105
Gateway 123.45.25.254
Subnet Mask 255.255.255.0

Static Ip 123.45.24.122
Gateway 123.45.24.254
Subnet Mask 255.255.255.0

Static Ip 123.45.24.123
Gateway 123.45.24.254
Subnet Mask 255.255.255.0

Sales Server 192.168.0.88 (only needs ports 80, 443, 8443) (ideally using public ip 123.45.25.105)

CentOS Server 192.168.1.75 (needs pretty much all ports) (ideally using public ip 123.45.24.122)

The CentOS server sends mail so it MUST use the same ip address for outgoing connections. I suppose I could add all the ip addresses to the spf record if absolutely necessary. But I would prefer to keep it all to one public ip address.

As far as every other device on the network, ideally I would like all traffic for that to use the third unused ip address, but I don't overly care about that part if it's a problem.

[landonious]

Yet my windows server that is using the other IP in that same subnet has zero issues.

Windows: 123.45.24.123 (zero issues)

CloudLinux: 123.45.24.122 (can't get out)

The only other difference between the two that I can think of is that the Windows server is only forwarding ports 80, 443, and 8443.

[pminarik]

There's no reason to be guessing.
Run a packet capture on the FortiGate to get an idea of what's going on.

Assuming the policy with VIP does not have SNAT enabled, the ideal filter will be:

diag sniffer packet any "host <client-ip> and port <VIP-extport>" 4 0 a

Alternatively:

diag sniffer packet any "host <client-ip> and (port <VIP-extport> or <VIP-mapped-port)" 4 0 a

If this is a public client, make sure to use the client's public IP.

If you don't even see an incoming SYN packet, the problem is either elsewhere, or you used a bad filter.

If the SYN packet arrives, pay close attention to whether it's seen forwarded towards the real internal server via the correct interface. If not, then you can switch to checking with debug flow to find out what happened to it.

[landonious]

I keep trying to reply with the output but it keeps removing my reply. Basically, when I try to connect with my phone it says multiple lines of:

[timestamp] wan1 in [my phone public ip] -> 123.45.24.122.2087: syn [numbers]

no ack on any lines

when i try to connect (successfully) from a local PC connected to the same network, it has several differing lines of output:

_default in

_default out

fortilink out

b out

And as I stated previously, the server itself cannot ping anything outside the local network. Not even 8.8.8.8. It's like all outgoing requests hit the FortiGate and then die. But the incoming appears to be working.

[pminarik]

You can try adding the outputs as a file attachment, maybe that will help?

Fort the phone scenario, run debug flow for the same traffic.

diag debug flow filter clear

diag debug flow filter addr <client-pub-ip>

diag debug flow filter port <destination-port>

diag debug enable

diag debug flow trace start 10

For the server unable to reach outside its subnet, you can do similar debugs (packet capture, debug flow).

This may also prove helpful: https://community.fortinet.com/t5/FortiGate/Troubleshooting-Tip-First-steps-to-troubleshoot-connecti...

[landonious]

I went ahead and gave up on this. I was just informed last night that our IP addresses will be changing when they bring in our fiber connection in January and they promised me they will all be in one block on the same subnet. However, before I received that information, I had sort of given up and gone a different route. Since the one IP was in a different subnet, I setup a second WAN using that one and then used policy routing to force it to use that connection and set the gateway there. Everything WAS working last night, but when I got in this morning and tried to change the IP address on that second WAN, it all stopped working (even worse than before) and I didn't want to try restarting since we have some people currently on web meetings. So I'm just giving up for now and going back to the way it was and I'll switch everything around when we get the fiber connection. I've got the windows server behind the FortiGate and it's really the only one I'm concerned about. Thank you guys for all your help.

[landonious]

So the problem ended up being that (per the guide below), I had set the second static route with an administrative distance of 20, whereas (according to support) I was supposed to have it at 10 like the first one and then set the priority with a higher number. I found this out, because the windows server stopped working, too, and I had to contact support to get it back up.

https://docs.fortinet.com/document/fortigate/7.4.1/administration-guide/360563/dual-internet-connect...

[landonious]

Going to try logs a different way.

When trying to use my phone's cellular data:

router # diag sniffer packet any "host 123.45.24.122 and port 2087" 4 0 a
interfaces=[any]
filters=[host 123.45.24.122 and port 2087]
2023-11-15 19:29:53.173770 wan1 in 172.56.52.20.42053 -> 123.45.24.122.2087: syn 1139511685
2023-11-15 19:29:53.173875 wan1 in 172.56.52.20.34953 -> 123.45.24.122.2087: syn 2558850117
2023-11-15 19:29:53.453702 wan1 in 172.56.53.180.31620 -> 123.45.24.122.2087: syn 2396595842
2023-11-15 19:29:53.453708 wan1 in 172.56.53.180.7878 -> 123.45.24.122.2087: syn 499275239
2023-11-15 19:29:54.165290 wan1 in 172.56.52.20.34953 -> 123.45.24.122.2087: syn 2558850117
2023-11-15 19:29:54.174547 wan1 in 172.56.52.20.42053 -> 123.45.24.122.2087: syn 1139511685
2023-11-15 19:29:54.494519 wan1 in 172.56.53.180.7878 -> 123.45.24.122.2087: syn 499275239
2023-11-15 19:29:54.494550 wan1 in 172.56.53.180.31620 -> 123.45.24.122.2087: syn 2396595842
2023-11-15 19:29:56.191501 wan1 in 172.56.52.20.42053 -> 123.45.24.122.2087: syn 1139511685
2023-11-15 19:29:56.191531 wan1 in 172.56.52.20.34953 -> 123.45.24.122.2087: syn 2558850117
2023-11-15 19:29:56.474520 wan1 in 172.56.53.180.31620 -> 123.45.24.122.2087: syn 2396595842
2023-11-15 19:29:56.474584 wan1 in 172.56.53.180.7878 -> 123.45.24.122.2087: syn 499275239
2023-11-15 19:30:00.194547 wan1 in 172.56.52.20.62405 -> 123.45.24.122.2087: syn 1139511685
2023-11-15 19:30:00.194591 wan1 in 172.56.52.20.38928 -> 123.45.24.122.2087: syn 2558850117
2023-11-15 19:30:00.504469 wan1 in 172.56.53.180.35224 -> 123.45.24.122.2087: syn 2396595842
2023-11-15 19:30:00.504497 wan1 in 172.56.53.180.1094 -> 123.45.24.122.2087: syn 499275239

When using a local PC on the same network:

router # diag sniffer packet any "host 123.45.24.122 and port 2087" 4 0 a
interfaces=[any]
filters=[host 123.45.24.122 and port 2087]
2023-11-15 19:29:16.369783 _default in 192.168.1.73.53420 -> 123.45.24.122.2087: fin 474414931 ack 1744755125
2023-11-15 19:29:16.369816 _default in 192.168.1.73.53425 -> 123.45.24.122.2087: fin 902917584 ack 3028685653
2023-11-15 19:29:16.369953 _default in 192.168.1.73.53432 -> 123.45.24.122.2087: syn 3689149562
2023-11-15 19:29:16.370054 _default in 192.168.1.73.53433 -> 123.45.24.122.2087: syn 514470926
2023-11-15 19:29:16.370558 _default out 123.45.24.122.2087 -> 192.168.1.73.53432: syn 1410238308 ack 3689149563
2023-11-15 19:29:16.370558 _default out 123.45.24.122.2087 -> 192.168.1.73.53433: syn 2629915381 ack 514470927
2023-11-15 19:29:16.370564 fortilink out 123.45.24.122.2087 -> 192.168.1.73.53432: syn 1410238308 ack 3689149563
2023-11-15 19:29:16.370564 fortilink out 123.45.24.122.2087 -> 192.168.1.73.53433: syn 2629915381 ack 514470927
2023-11-15 19:29:16.370568 b out 123.45.24.122.2087 -> 192.168.1.73.53432: syn 1410238308 ack 3689149563
2023-11-15 19:29:16.370568 b out 123.45.24.122.2087 -> 192.168.1.73.53433: syn 2629915381 ack 514470927
2023-11-15 19:29:16.370743 _default in 192.168.1.73.53432 -> 123.45.24.122.2087: ack 1410238309
2023-11-15 19:29:16.370748 _default in 192.168.1.73.53433 -> 123.45.24.122.2087: ack 2629915382
2023-11-15 19:29:17.804226 _default out 123.45.24.122.2087 -> 192.168.1.73.53433: psh 2629927578 ack 514472621
2023-11-15 19:29:17.804235 fortilink out 123.45.24.122.2087 -> 192.168.1.73.53433: psh 2629927578 ack 514472621
2023-11-15 19:29:17.804240 b out 123.45.24.122.2087 -> 192.168.1.73.53433: psh 2629927578 ack 514472621
2023-11-15 19:29:17.804261 _default out 123.45.24.122.2087 -> 192.168.1.73.53433: fin 2629928926 ack 514472621
2023-11-15 19:29:17.804265 fortilink out 123.45.24.122.2087 -> 192.168.1.73.53433: fin 2629928926 ack 514472621
2023-11-15 19:29:17.804268 b out 123.45.24.122.2087 -> 192.168.1.73.53433: fin 2629928926 ack 514472621
2023-11-15 19:29:17.804278 _default in 192.168.1.73.53433 -> 123.45.24.122.2087: ack 2629927578
2023-11-15 19:29:17.804622 _default in 192.168.1.73.53433 -> 123.45.24.122.2087: ack 2629928927
2023-11-15 19:29:17.805260 _default in 192.168.1.73.53433 -> 123.45.24.122.2087: fin 514472621 ack 2629928927
2023-11-15 19:29:17.871215 _default in 192.168.1.73.53435 -> 123.45.24.122.2087: syn 464554507
2023-11-15 19:29:17.871752 _default out 123.45.24.122.2087 -> 192.168.1.73.53435: syn 3820942514 ack 464554508
2023-11-15 19:29:17.871758 fortilink out 123.45.24.122.2087 -> 192.168.1.73.53435: syn 3820942514 ack 464554508
2023-11-15 19:29:17.871761 b out 123.45.24.122.2087 -> 192.168.1.73.53435: syn 3820942514 ack 464554508
2023-11-15 19:29:17.871933 _default in 192.168.1.73.53435 -> 123.45.24.122.2087: ack 3820942515

[landonious]

I'm accepting this as the solution because it is indeed the solution in general. My scenario is just an odd one-off due to other issues that we haven't figured out.

[ede_pfau]

Exactly, use VIPs (DNAT) to "expose" your internal servers, not secondary IPs. The latter would just point to the FGT, not to any server.

If you set up the 2 policies correctly, outbound traffic (which is not reply traffic) will be re-translated to the public IP specified in the VIP.

Not using port translation has the by-effect that you'll lose that public IP completely for other purposes - it will act as the server's IP. So it should not be the one public IP you are using for IPsec VPN or such.

Finally, if you need to translate 3 ports, define 3 VIPs and group them into a VIP group.

[landonious]

When I move my cPanel server over to the FortiGate, it cannot access the internet. I set the internal IP to 192.168.0.78 (which is where the VIP points to) and nothing. I can ping every single device on the local network no issues, but I cannot access anything beyond. If I simply change the ip to 192.168.0.76 (which is not referenced anywhere in the FortiGate...just a random unused ip address) it can access everything just fine. My thinking is that since the public IP we are using for this one (123.45.24.122) is different from the WAN interface (123.45.25.205) and uses a different gateway that this might be the problem. Does that sound right? The only static route I have setup is for the 123.45.25.205 IP which is 123.45.25.254. If you think this is the problem, can we get around this? I'm not sure how to set another static route for this, or is that possible?

EDIT: of course, now that I've typed this, I realized that the windows server that is using ip 123.45.24.123 is not having any issues connecting to the internet. However, that VIP is setup to only forward 80, 443, and 8443. Does that change things?