Whenever I’ve got a lot of data to transfer I usually kick off the rsync or scp with the default SSH settings and then, after an hour or so, wonder if I could have sped things up with a different cipher.
So I decided to find out now for my future self and maybe save some time.
A quick not about SSH Ciphers
SSH uses several ciphers and algorithms. I have encountered lots of sysadmins that think their choice of SSH key determines their transfer speed. SSH keys are only used by SSH to transmit the shared key that is used for the symmetric cipher which encrypts all of the session data. The admins SSH key does not affect the transfer speed only the choide symmetric cipher does.
The cipher can be manually set when starting an SSH session using the -c <CIPHER> option.
The list of ciphers that your versions of SSH supports is printed with ssh -A ciphers. On my two Ubuntu 20.04 test servers this is:
# ssh -Q ciphers
3des-cbc
aes128-cbc
aes192-cbc
aes256-cbc
rijndael-cbc@lysator.liu.se
aes128-ctr
aes192-ctr
aes256-ctr
aes128-gcm@openssh.com
aes256-gcm@openssh.com
chacha20-poly1305@openssh.com
So these are the ones I’m going to test.
The Setup
First, all of these ciphers need enabling on both sides. I did this by editing /etc/ssh/sshd_config and adding the following line:
Ciphers 3des-cbc,aes128-cbc,aes192-cbc,aes256-cbc,rijndael-cbc@lysator.liu.se,aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm@openssh.com,aes256-gcm@openssh.com,chacha20-poly1305@openssh.com
Next, I generated a 500MB file of random data:
dd if=/dev/urandom of=rand.file bs=1MB count=500
Then, I spun up a couple of DigitalOcean droplets with a VLAN between them which I used for these tests.
Results
First, I tested the how fast the disk was:
dd if=rand.file of=/dev/null
Then how fast an unencrypted transfer between the servers. I used netcat for that as follows:
Sender:
dd if=rand.file | nc <IP> 8080
Receiver
nc 8080 -l >/dev/null
And finally, using SSH with the following command:
dd if=rand.file | ssh -o 'Compression no' -c <CIPHER> <USER>@<IP> dd of=/dev/null
These are the numbers I got:
| Speed | |
|---|---|
| Raw disk read | 183 MB/s |
| Unencrypted | 83.2 MB/s |
| 3des-cbc | 14.1 MB/s |
| aes128-cbc | 75.7 MB/s |
| aes192-cbc | 74.1 MB/s |
| aes256-cbc | 68.9 MB/s |
| rijndael-cbc@lysator.liu.se | 64.6 MB/s |
| aes128-ctr | 67.4 MB/s |
| aes192-ctr | 69.5 MB/s |
| aes256-ctr | 68.2 MB/s |
| aes128-gcm@openssh.com | 65.3 MB/s |
| aes256-gcm@openssh.com | 65.0 MB/s |
| chacha20-poly1305@openssh.com | 67.0 MB/s |
The CPU was at or around 100% during all of these transfers so it was all down the the efficiency of the ciphers.
The winner is aes128-cbc. So if you are looking to speed up a transfer choose that one. Here are the commands for rsync:
rsync -e `ssh -c aes128-cbc` rand.file user@example.com:/home/user/rand.file
And scp:
scp -c aes128-cbc rand.file user@example.com:/home/user/rand.file