Issiue with Fortigate HA/Fortiswitch HA configuration

[PicoloSan]

Hello

From few days I'm in company where they build network on fortigate devices, before I work only with mikrotik solution, so i need to understand little bit the "fortios idea" So is good time becouse from saturday we notice strange sitauation.

In this company they have two localization from one to second is some 25 kilometers, they have direct fiber connection between. This fiber cable are connected to Fortiswitch on site A port 51, on site B port 52.

To both Fortiswitch to port1 is connected ISP also with SD-WAN configuration

Site A, Fortiswitch Port1 IPS X1 (vlan_X1 -> Fortilink)

Site B, Fortiswitch Port1 IPS X2 (vlan_X2 -> Fortilink)

On fortigate devices the DMZ port is used for HB, so from both Fortigates DMZ port is connected to port 20 in both Fortiswitch. HA type is A-P, Also SD-WAN

This is small picture how this looks with physical connection, and this issiue we have

On saturday fiber between Site and Site was broken, SD-WAN showed on both fortigates the "Second" link is unreachable, but no internet connection from hosts.

After fiber connection go up, i did some test, and this looks like that:

1. When WAN1 is disconnected everything works, Hosts have internet service on both Sites with WAN2.

2. When WAN2 is disconnected everything works, Hosts have internet service on both Sites with WAN1.

3. When Fiber is disconencted no internet conenction on both Sites

From info I have this works... in last year, No one know what configuration was ok, but i know the update was done many times, and some changes in cfg was done also. But never tested...

I read in fortinet documentation and check few network topology, and no one have in description the situation is here, where WAN are connected to Fortiswitch almost evrywhere WAN is connected to Fortigate ports (but I dont think this is the issiue).

Also I see almost everywhere in doc, where I can see two Sites, isinfo about MCLAG on Fortiswitch, here in cfg I dont see this.

Is someone on this forum who can share his expirence with this solution, or give me link where i can found some info i documentation how to configure HA, and Forlink/SwitchController to resolve issiue.

Regards

[adambomb1219]

Not enough information here really.  What "fiber" is disconnected?  The link between the two firewalls?

[PicoloSan]

Like on the picture, there is only one Fiber connection, this is 25 kilometers link between two localization connected to Fortiswitch Site A Fortiswitch Port 51 - Site B Fortiswitch Port 52

Between two Firewall (Fortigate) any connection not exist (I mean direct), the everythink like SD-WAN, HA,  Physical connection from ISP (WAN) etc are conencted to ports in Fortiswitch. Form Firewall (Fortigate) only port A&B are connected to Fortiswitch nothing else

[adambomb1219]

Wait so you are running HA southbound to the FortiSwitch and then over a single fiber?  That is not the correct way to setup HA.  If that is the case these firewalls should be standalone devices and use an internal dynamic routing protocol instead.

[PicoloSan]

To clarify more the toplogy looks like  this seciotn

HA-mode FortiGate units in different sites

In document Deploying MCLAG topologies | FortiSwitch 7.4.2 | Fortinet Document Library

But only with One Fortiswitch on every site, and WAN not connected directly to Fortigate (Firewall) but to Fortiswitch

[Toshi_Esumi]

Of course I've never set FGT HA remotely and user traffic & HA traffic on the same link. But based on your description, your intended operation when the 25Km fiber goes down seems to be:
- FG01 and FG02 would become "Primary" since they can't see each other
- FG01 uses WAN1 for the internet and the SiteA users use the internet path
- FG02 uses WAN2 for the internet and the SiteB users use the internet path

Based on that, First thing you need to check this the HA status on both units when the fiber is down.
"get sys ha status"
Make sure both are "primary" and the SN on the other side shouldn't show in the output. I assume it does.
Then, the second thing you need to check is the internet connection on both sides. Just get in each FGT and ping out to the internet. If SD-WAN is working properly, you should be able to ping. I would assume this works too.

And then, the third thing you need to check is the LAN side connectivity. Check if you can ping the user devices from the FGT on the same site when the fiber is down. If not, check ARP table if it can see devices (or ARP timers increment). My guess is this part is likely the issue.
Sounds like you're using MCLAG, which I never tried. So, if you confirmed above and MCLAG seems to be the issue, you can ask a troubleshooting methods of MCLAG in this thread. Or better, open a TAC ticket to get it looked at.

Toshi