Load test packet loss

[theoleek]

I am try to load test FortiGate VM using cisco Trex. While this works perfectly on the current pfsense setup, I cannot get it to work on FortiGate.

Simple routing and firewall rules have been set up the same as pfsense and  can see that the one firewall rule is beng used based on the bandwidth usage. When looking at foward traffic logs it seems that some traffic is getting through fine but the majority of traffic does not seem to be accepted as is mostly droppped.

I have tried the following so far;

• Using Policy routes rather than static routes produces the same result
• Change the interface types from unspesified to WAN or LAN
• Added DoS policy, no change
• Changed NAT settings in firewall rules
• Changed protocol options in firewall rules

Looking at forward traffic it seems that traffic that is having issues is the following

Duration	5
Session ID	63,877
VDOM	root
NAT Translation	noop

Source

Source	16.0.0.14
Source Port	5,796
Source Country/Region	United States
Primary Source Mac	00:0c:29:93:42:f3
Source Interface

port1

Destination

Destination	48.0.7.7
Destination Port	80
Destination Country/Region	United States
Destination Interface

port2

Application Control

Application Name	HTTP
Category	unscanned
Protocol	6
Service	HTTP

Data

Received Bytes	33.02 kB
Received Packets	23
Sent Bytes	813 B
Sent Packets	14

Action

Action	client-rst
Security Action
Policy ID	0to1
Policy UUID	57417294-aca1-51ed-d32e-e59d083a0abd
Policy Type	Firewall

Security

Level

notice

Cellular

Service

HTTP

Other

Log event original timestamp	1676410894150044700
Timezone	+0000
Log ID	0000000013
Type	traffic
Sub Type	forward
Source Interface Role	wan
Destination Interface Role	lan
Policy Name	0to1
Source Server	0

[adambomb1219]

Is the VM licensed?  Any Inspection profile configured in the firewall policy?

[theoleek]

The VM has a free lisence and is registered yes. Here is the fireall rules and not inspection or security rules are used.

port1 →port2

1to0

all

all

Disabled

ACCEPT

All

always

ALL

no-inspection

18.64 MB

port2 →port1

0to1

all

all

Disabled

ACCEPT

All

always

ALL

no-inspection

0 B

[gfleming]

Source IP 16.0.0.14. Do you own this address? If not it might not get routed back to you properly.

Can you further explain the actual flows you are trying to get working. Does the FortiGate connect to an actual ISP or is this all internal testing? What kind of traffic does trex generate? Does it all go to the same destination or multiple?

[theoleek]

Thank you for having a look Graham. So the setup is as follows.

TREX port0-------->FortiGate port1         FortiGate port2 -------->TREX port1

11.11.11.10/24                 11.11.11.20/24              12.12.12.20/24                   12.12.12.10/24

Routes are

16.0.0.0/8

11.11.11.10

port1

Enabled

48.0.0.0/8

12.12.12.10

port2

Trex sends packets with Source IP 16.0.0.0/8 and destination 48.0.0.0/8 out of port0

It then sends Source IP 48.0.0.0/8 and destination 16.0.0.0/8 out of port1

Cheers

[gfleming]

Are all of the log messages showing action of "client-rst"?

This implies there is a TCP Reset being sent by the client, or the source, for that TCP session.

Can you check your logs and see if you see any other actions?

[theoleek]

Some packets seem to be fine, for example

Source

Source	16.0.0.13
Source Port	53,921
Source Country/Region	United States
Primary Source Mac	00:0c:29:93:42:f3
Source Interface

port1
OS Name	Windows

Destination

Destination	48.0.1.12
Destination Port	80
Destination Country/Region	United States
Destination Interface

port2

which comes back as

Action

ip-conn

[gfleming]

Considering this is a VM, have you ensured your VM Guest config is accurate and properly set up for the FortiGate-VM? 

Have you confirmed connectivity from the FortiGate to both sides of the Trex appliance?

From FortiGate CLI can you ping out from both port1 and port2 to either port on the Trex?

"execute ping <IP address>" is the CLI command to run.

[theoleek]

No, the FortiGate is unable to ping the TREX ports and TREX is unable to ping the FortiGate ports. This is the same behaviour with pfsense. When TREX first starts it sends ARP packets to the FortiGate ports which works as TREX then starts sending traffic.

This is setup on ESXi 6.5 using VMXNET3 adaptors and is using 2 seperate virtual switches for the connections between them, promisucous mode enabled.

I have reinstalled a fresh copy of FoirtGate and simply added interface IP's, added static routes, and added the firewall policy and still getting the same result. TREX reports packets drop after 10 seconds of test traffic.

[gfleming]

What version of FortiOS are you running?

And you've confirmed compatibiliy? https://docs.fortinet.com/document/fortigate-private-cloud/7.2.0/vmware-esxi-administration-guide/23...