Load test packet loss

[theoleek]

I am try to load test FortiGate VM using cisco Trex. While this works perfectly on the current pfsense setup, I cannot get it to work on FortiGate.

Simple routing and firewall rules have been set up the same as pfsense and  can see that the one firewall rule is beng used based on the bandwidth usage. When looking at foward traffic logs it seems that some traffic is getting through fine but the majority of traffic does not seem to be accepted as is mostly droppped.

I have tried the following so far;

• Using Policy routes rather than static routes produces the same result
• Change the interface types from unspesified to WAN or LAN
• Added DoS policy, no change
• Changed NAT settings in firewall rules
• Changed protocol options in firewall rules

Looking at forward traffic it seems that traffic that is having issues is the following

Duration	5
Session ID	63,877
VDOM	root
NAT Translation	noop

Source

Source	16.0.0.14
Source Port	5,796
Source Country/Region	United States
Primary Source Mac	00:0c:29:93:42:f3
Source Interface

port1

Destination

Destination	48.0.7.7
Destination Port	80
Destination Country/Region	United States
Destination Interface

port2

Application Control

Application Name	HTTP
Category	unscanned
Protocol	6
Service	HTTP

Data

Received Bytes	33.02 kB
Received Packets	23
Sent Bytes	813 B
Sent Packets	14

Action

Action	client-rst
Security Action
Policy ID	0to1
Policy UUID	57417294-aca1-51ed-d32e-e59d083a0abd
Policy Type	Firewall

Security

Level

notice

Cellular

Service

HTTP

Other

Log event original timestamp	1676410894150044700
Timezone	+0000
Log ID	0000000013
Type	traffic
Sub Type	forward
Source Interface Role	wan
Destination Interface Role	lan
Policy Name	0to1
Source Server	0

[theoleek]

Ive tested a few different combinations of guest os types and still gettng same result not matter what the test speed is

[theoleek]

When i enable NAT in the firewall rules then TREX report packet loss straight away but forward log show all as accepted rather than being blank like it is with NAT turned off.

[theoleek]

Packet Trace #1

12:48:50

vd-root:0 received a packet(proto=6, 16.0.0.1:41668->48.0.0.1:80) tun_id=0.0.0.0 from port1. flag [S], seq 404375002, ack 0, win 32768

12:48:50

allocate a new session-0000292a, tun_id=0.0.0.0

12:48:50

in-[port1], out-[]

12:48:50

len=0

12:48:50

result: skb_flags-02000000, vid-0, ret-no-match, act-accept, flag-00000000

12:48:50

find a route: flag=00000000 gw-12.12.12.10 via port2

12:48:50

in-[port1], out-[port2], skb_flags-02000000, vid-0, app_id: 0, url_cat_id: 0

12:48:50

gnum-100004, use int hash, slot=30, len=2

12:48:50

checked gnum-100004 policy-1, ret-matched, act-accept

12:48:50

ret-matched

12:48:50

policy-1 is matched, act-accept

12:48:50

after iprope_captive_check(): is_captive-0, ret-matched, act-accept, idx-1

12:48:50

after iprope_captive_check(): is_captive-0, ret-matched, act-accept, idx-1

12:48:50

Allowed by Policy-1:

12:48:50

npu_state=0x100, hook=4

[theoleek]

Packet Trace #2

5

12:48:50

vd-root:0 received a packet(proto=6, 48.0.0.1:80->16.0.0.1:41668) tun_id=0.0.0.0 from port2. flag [S.], seq 404419109, ack 404375003, win 32768

12:48:50

Find an existing session, id-0000292a, reply direction

12:48:50

find a route: flag=00000000 gw-11.11.11.10 via port1

12:48:50

Trying to offloading session from port2 to port1, skb.npu_flag=00000400 ses.state=00000200 ses.npu_state=0x00000100

12:48:50

state=00000200, state2=00000000, npu_state=00000100

[theoleek]

Packet Trace #3

3

12:48:50

vd-root:0 received a packet(proto=6, 16.0.0.1:41668->48.0.0.1:80) tun_id=0.0.0.0 from port1. flag [.], seq 404375003, ack 404419110, win 32768

12:48:50

Find an existing session, id-0000292a, original direction

12:48:50

enter fast path

[theoleek]

Above is a few of the debug packets for address 16.0.0.1 when starting load test if this provides any aditional information

[theoleek]

Definatly some sort of issue with FortiGate. Ive gone through may settings combinations over the last 7 days, even tried on an older version of FortiGate and always the same result. Works for about 10 seconds then packet loss. With the Forward traffic result column empty, client rst packet drop is the only thing I can identify as a potential issue. Have tried both set timeout-send-rst enable and set timeout-send-rst disable

It should be simple enough, set up interface IP's, set up routing, set up firewall rules. I have tried this with SonicWall and it works perfectly first time.

Im just at a total loss for anything to try next as I am unable to resolve after days of research and going through different settings.

If anyone has any other sugestions on what could be wrong i my setup I'd be increadable greatful.

Cheers

[vsahu]

Can you disable the offloading in the firewall policy and check?

config firewall policy

edit x

set auto-asic-offload disable

Also, share the output of the session table after initiating the traffic, with the address list output.

diag sys session filter src <source IP address>

diag sys session list

diag ip address list

[theoleek]

Same result sadly. The output seems to be the same for src ip 11.11.11.10 and 16.0.0.1

This is the TREX output, 0.00Mbps drop-rate until exactly 10 seconds

[theoleek]

Hi Vishal,

The forward traffic results are still blank for most of the logs in the result column with most packets having this entry

Received Bytes	33.02 kB
Received Packets	23
Sent Bytes	813 B
Sent Packets	14