Visitor III

Question

Problem with FGSP and FGCP

Forum|Forum|3 years ago
February 21, 2023
24 replies
9351 views

Hello everyone,

FortiOS: 7.2.4

Fortigate: 200E

We have two FGCP clusters and FGSP between them. FGCP clusters are georaphically spaced and RTT between them around 40-50 ms. Session sync is configured over L3 link between FGCP clusters.

We have configured pickup sessions(also expectation and connectionless).

1st FGCP cluster:

config system ha
set group-name "cluster 01"
set mode a-p
set session-pickup enable
set session-pickup-connectionless enable
set session-pickup-expectation enable
set ha-mgmt-status enable
set override disable

config system standalone-cluster
set standalone-group-id 1
set group-member-id 1
config cluster-peer
edit 1
set peerip x.x.x.x

diagnose sys ha standalone-peers
Group=1, ID=1
Detected-peers=1
Kernel standalone-peers: num=1.
    peer0: vfid=0, peerip:port = y.y.y.y:708, standalone_id=2
    session-type: send=249986, recv=403283
    packet-type: send=0, recv=0

2nd FGCP cluster:

config system ha
set group-name "cluster 02"
set mode a-p
set session-pickup enable
set session-pickup-connectionless enable
set session-pickup-expectation enable
set ha-mgmt-status enable
set override disable

config system standalone-cluster
set standalone-group-id 1
set group-member-id 2
config cluster-peer
edit 1
set peerip y.y.y.y

diagnose sys ha standalone-peers
Group=1, ID=2
Detected-peers=1
Kernel standalone-peers: num=1.
peer0: vfid=0, peerip:port = x.x.x.x:708, standalone_id=1
session-type: send=202291, recv=4528433
packet-type: send=0, recv=0

Sessions are synchronized without problem.

So, problem is following:

When traffic symmetrical we have no problem. Symmetrical traffic mean that traffic came out from and came back to the same FGCP cluster (for example 1st FGSP cluster).

But when traffic asymmetrical we have problem: using icmp as an example we have lost first or two packets. Using TCP we have long connection, for example, to smtp services. Using UDP, for example, DNS server sometimes has timeout error. Asymmetrical traffic mean that traffic came out from one FGCP cluster and came back to another FGCP cluster (for example came out from 1st FGSP cluster and came back to 2nd FGSP cluster). So, we have this problem with both TCP and UDP.

For now we have following investigation results:

1) This problem is not related to traffic inspection and observed on both type of rules: with traffic inspection and without traffic inspection.

2) We don't observe this problem when traffic just go through FGCP clusters without NAT.

3) We don't observe this problem when traffic symmetrical.

4) Session synchronization occurs instantly with first packet on one of the FGCP cluster.

So, I suppose problem with NAT. But actually how can I debug this? Maybe some tuning options exist? Has someone encountered such a problem?

FortiGate

srajeswaran

Staff

NAT sessions are not synchronized by default in FGSP. You can enable NAT session synchronization by entering the following command:

config system ha

set session-pickup enable

set session-pickup-nat enable

end

You need a common pool as well.

Ref:

https://docs.fortinet.com/document/fortigate/6.4.0/ports-and-protocols/796662/fgsp-fortigate-session-life-support-protocol

maxiboomAuthor

Visitor III

@srajeswaran

In ForiOS 7.2.4 command "set session-pickup-nat enable" is missing and as I understood

it's enough to use command "set session-pickup enable" to sync NAT session. Moreover I have seen a session replica with status syn_ses on neighbor cluster.

srajeswaran

Staff

It is available in 7.2.4 and is required as well along with the pool.

https://docs.fortinet.com/document/fortigate/7.2.4/administration-guide/18108/optimizing-fgsp-session-synchronization-and-redundancy

# get system status
Version: FortiGate-VM64 v7.2.4,build1396,230131 (GA.F)

(ha) # show
config system ha
set session-pickup enable
set session-pickup-nat enable
set override disable
end

srajeswaran

Staff

So packets are matching the session, but not going out. Isn't it?

I think its better to open a TAC ticket (could be a bug).

srajeswaran

Staff

Can you share the complete debug flow from both FGSP members.

maxiboomAuthor

Visitor III

Screenshots above are complete five ping or what do you mean?

AEK

SuperUser

Hello

Any fix for this issue?

AEK

SuperUser

Hi

I've faced something similar for UTM sessions only, and could finally fix it by tuning the below params:

FGSP configuration -> unset session-sync-dev
HA configuration -> set sync-packet-balance enable
UTM policy -> set auto-asic-offload disable

Hope this helps

AEK

RenataK

Visitor III

Hello there.

Any fix for this issue? Is it a bug ?

AEK

SuperUser

Many FGSP related issues have been fixed since 7.2.4. Try update to 7.2.8 and see if it helps.

AEK

RenataK

Visitor III

The weird thing is that not all system are impacted. Our fortigate are in multi-VDOM mode and only one is impacted but for some resource only (like Citrix)

Here are some investigation results:

We can see that sessions are synchronized.
We tried to disable/enable UTM feature, anti-replay and tcp-session-without-syn with no success.
We don't observe this problem when traffic just go through one firewall (traffic symmetrical) .

Sign up

Login with SSO

Login to the community

Login with SSO