|
The reason transferring a compressed file (for example ZIP) is faster than transferring a folder containing multiple files is because of overhead and latency.
When transferring a folder the negotiation is done for each file, meaning transferring N files the negotiation is to be performed N times. While for transferring a compressed file the negotiation is only performed once.
With read/write multiple files the system introduces overhead too, as it needs to read/write the directory for each file. For a file system, the system needs to check things file names and permissions and write out the file table entry. Each of these steps adds latency to a copy/transfer process. The more files the longer it takes.
Packet size is also relevant here. With lots of small files, the system can send into the network multiple small packets. While large files data are usually sent in fewer but larger packets, becoming more efficient.
In a summarized way, transferring a compressed file is faster because it bundles data minimizing file system and network overhead avoiding repeated negotiations related to individual file transfer, and more so when applying encryption and decryption.
The below images display the throughput difference and provide a real scenario example of transferring a folder containing multiple files versus transferring the same folder compressed into a compressed file.
Transferring folder containing multiple files:
Transferring the same folder but compressed into a ZIP file:
Breakdown of Key Points:
File System Overhead.
When transferring multiple files, the operative system must process metadata (for example: file names, permissions, timestamps, etc) for each file. For each file, the system has to:
- Check directory entries.
- Verify if the file name is unique.
- Write updates to the file system index.
A compressed file consolidates all files into one, resuming all these operations into a single operation.
Network Protocol Overhead.
For each file transfer in a folder, the network protocol (for example: FTP or SMB) initiates a handshake and acknowledgment process. Transferring multiple files results in multiple negotiations.
Packet Size and Efficiency.
Transferring multiple small files can result in the network sending multiple smaller packets. And smaller packets incur a higher number of packets leading to higher TCP/IP overhead due to headers and acknowledgments, even more so when applying encryption and decryption.
Caching and Disk Operations.
Reading and writing a compressed file typically takes advantage of sequential disk access, which is generally faster than the random access used for reading and writing multiple files.
Notes:
SMB should not be used for WAN file transfers (for example: IPsec scenarios).
FortiGate Protocol Options allow to forcefully change TCP window size to improve the throughput for certain protocols (for example CIFS = SMB, which uses port 445), but this is just a workaround. The proper solution requires to review of the environment network architecture and using a different protocol.
|
