How to speed up copying files over the network

Assuming that ftp service is disabled for security reasons, the most admins are using scp to copy files to/from the target server.

As many users have already noticed, the file transfer is slow (at least slower then NFS share or Samba for example).

Reason for this is in the letter “s” in the scp command which comes from the secure copy.

SCP first encrypts all the data before sending, transfer files which has to go through decryption on the target machine.

Security is of critical importance when you are sending files over open network, and default cipher is usually enough.

On the other side, when you have to transfer files over a local network, you can use weaker encryption algorithms.

SSH (mechanism behind scp command) has several ciphers you can choose, depending on your encryption strength requirement.

It’s clear when you have to transfer large installation files (like Oracle database) over a local network, there is no need for strong encryption.

In the following examples I’m going to compare default scp cipher and acrfour which is using weaker encryption algorithm.

Test 1 – Using the default cipher:

test@test.acme.local:/tmp> time scp app.zip test@test.test.com:/tmp

app.zip 100%

real 1m6.814s
user 0m10.509s
sys 0m2.159s

Test 2 – Using the weaker cipher:

test@test.acme.local:/tmp> time scp -c arcfour app.zip test@test.test.com:/tmp

app.zip 100%

real 0m45.218s
user 0m6.060s
sys 0m1.978s

If you get a ‘no matching cipher found’ error, make sure the ciphers you use are in the ciphers line of /etc/ssh/sshd_config on both systems.

In this case I didn’t measure CPU consumption (especially on the sender machine), which would be greater in the first test.

As expected, real time (wall clock time) and user time (amount of CPU time spent in user mode) is much lower in the second case, as the encryption algorithms is weaker.

In case you have more than one file to copy, you can speed up the whole process even further by creating bash script that will copy files in parallel.

To conclude, in all cases when you are on private network and you are not sending sensitive data over the network, It’s safe enough to use weaker encryption when copying files over the network, especially If you are sending large files like several Gb of data.

Get notified when a new post is published!

Name*

Email*

 Please accept terms & conditions