Slow Veeam Backups through 2x fortigates

[dangelov]

Hi All,

We have FortiGate 1101E (6.4.13) and FortiGate 600F (v7.0.12), connected via a dark fibre (10 gig).

Before it was all switches with dark fibre and the firewalls will only inspect the Internet traffic, but now the firewalls take care of the routing too. No VDOMs or Virtual Clusters.

It was suggested to me that the internal architecture of the firewalls might be limiting us, how do I check this?

Migrating the routing from the switches to the firewalls was done in stages. The 600F was the latest to be introduced and after that the infrastructure team started complaining about slow back-ups, 3 to 3.5 times slower (3h to 4hours), all the back-ups are going to the site with the 600f.

No inspection on the packets.

No CPU or SPU issues. All the sessions run on hardware.

The 10 gig link touches the 6 gig usage during backups, but not constantly. Probably start with 1gig usage, grows to 3 and 4 gig more constant, burst to 5 and 6 on occasions.

I have created a little drawing, and I will try to explain to the best of my knowledge how the back-ups work.

DC1, DC2 and Back-Up Storage are different sites at different town location connected via dark fibre.

Backups are automated and start at night when there isn't much traffic.

I think 2 or 3 devices on subnet 10.0.12.x (residing on router) gather data from subnets 10.0.16.x, 10.0.60.x residing on DC1-FortiGate 1101E and 10.1.16.x residing on DC2-FortiGate 1101E.

A device on subnet 10.0.12.x (at DC1) will forward the data to a device on subnet 10.6.98.x at Back-UP Storage.

I observed the process, the transfer between DC1 and Back-Up Stoarge had multiple session between only 2 devices, one on 10.0.12.x (DC1) and one on 10.6.98.x at Back-UP Storage.

From the start there was one session that started with over 100 GB data and eventually grew to over 800 GB. A second session started about 30 minutes later with again over 100 gig and the data grew too. All other sessions were small data amounts, probably heartbeats or something, not sure how Veeam works. All sessions run on hardware on both firewalls.

The devices on 10.0.12.x are connected via 2x10 Gig active link. The subnets 10.0.16.x, 10.0.60.x use the 2x 10Gig link between the Firewall and the Router.

Sorry for the long post. Thanks in advance.

[AEK]

Hello

In general when a high backup traffic transits through 2 or 3 firewalls it will slow down for sure.

If this can be useful for you, know that in 95% of the companies I worked with, we always avoided transiting backup data through firewalls, but through closed networks dedicated to backup, not accessible by users and even usually not routable. I guess you should agree that this is one of good ways to secure backup data and traffic without exhausting firewalls.

[dangelov]

Hi AEK,

Thanks for taking the time to reply.

Yes, what you are suggesting is a good solution for one site or campus, but in my case I have 3 separate site locations, 3 different towns.

Yes, we have dark fibre between them but I will be compromising on security and I will only bypass one of the firewalls. 

I am just going to ask the same question again "It was suggested to me that the internal architecture of the firewalls might be limiting us, how do I check this?"

Thanks,

David

[AEK]

@dangelov , I guess they mean whether the related policy is flow mode or proxy mode, is using security profiles or not, traffic shaping ...etc. And/or it is about firewall performance, ASIC, sizing, ...etc

[dangelov]

Hi AEK,

@AEK  - I cannot ask the guy again as he was a consultant when I worked for a different company. 

I think it was more about firewall limitations, as I told him about the policies and that there is no inspections, flow-based, etc

Thanks