Rsync – Best Tool To Sync Files Between Devices (Mainly Servers)
rsync is a Linux CLI to copy or sync files between location A and B where location A and B can differ from network, host or device type. It is used at most by system admins to copy or move files through the network and even works on local files. rsync uses a delta-transfer algorithm to transfer files, reducing the time it takes to send files remotely and locally with tags available to only incrementing the remaining missing part of files.
Its algorithm (lqquickcheckrq algorithm) includes checking for files that have a recent modified date and copy only these new files.
What rsync can do
According to rsync man page, rsync has some good features that follow below:
- It allows users to copy links, devices, owners, groups, and permission from one host to another or between folders;
- Exclude and exclude-from options like GNU tar;
- A CVS exclude mode for ignoring the same files that CVS would ignore;
- Can use any transparent remote shell, including ssh or rsh;
- Does not require super-user privileges;
- Pipelining of file transfers to minimize latency costs;
- Support for anonymous or authenticated rsync daemons (ideal for mirroring).
How to use rsync to copy files
rsync allows a user only to copy files from or to remote hosts if either you are the local host (host A) or the remote host (host B) and not a third host (host C) connected, pretending to transfer from A to B.
rsync has too many commands you can see on its man page. Today we are going to show you the most commonly used commands for starting right away.
As we already know, good explanations are better with examples, and we will show you below how would you send a video called tutorial.avi from our ubuntu laptop to linuxandubuntu server on the folder videos located on the home folder.
rsync -t tutorial.avi linuxandubuntu:~/videos
Note that you can copy all files with the same extension using wildcards. Suppose we have more than one file to be transferred with the same extension, we can use the code below:
rsync -t *.avi linuxandubuntu:~/videos.
The code above will copy all our videos with extension avi to our videos folder in our home directory. rsync commands locally are more like cp and mv commands which means that you can use it also not just for backing up data, but also to keep them organized or move them from origin folder.To copy files from a server on a folder called videos to a local folder on our computer named tutorials we can use the code below.
Please note that the wildcard ‘~‘ means that the folder is located in
home/user/tutorials rsync -t linuxandubuntu:~/videos/*.avi ~/tutorials/
Let us now copy files locally from a folder named videos to tutorials on an external Hard Disk Drive (HDD). The videos folder has many types of videos extensions. We have a Big Bug Bunny.mp4, tut01-installing ubuntu.mp4, and other files. all our tutorials names start withtut but, not all have an mp4 extension, some are avi or mkv. if we use the wildcard *.mp4 we would copy all the files with mp4 extension including Big Bug Bunny.mp4 and other video files not important now and miss some tutorial videos. Our goal now is to only copy tutorial videos to external HDD. To do that we can use:
rsync tut-* /media/hdd/tutorials
for some reasons, we lost power on our computer and the sync was interrupted, to continue from where it stopped, use the command below:
rsync -P --update tut-* /media/hdd/tutorials
Noted the -P on the code? This is to show the progress in a percentage of what is being transferred and it will show only the percentage of a single file and not the overall percentage. The command above will ignore already transferred files and proceed with syncing.
rsync tut-* /media/hdd/tutorials
Syncing between folders
For folder transfer, we can use two different approaches. The first approach would copy the files inside the folder videos to location tutorials and ignore its parent folder (videos folder).
rsync -rP --update ~/videos/ /media/hdd/tutorials
The second method will copy the files into the folders and its parent folder.
rsync -rP --update ~/videos /media/hdd/tutorials
Note that in both cases we use -r to copy directories recursively and also note the absence of slash (/) on the second example for copying files with parent folder.
rsync is a good way to keep our files up-to-date. It is better than cp or mv because you can mirror the exact files on both sides. You can also compress files on transfer if needed. Give rsync a try and live your comments below.