How To Synchronize Directories with Rsync
Today I’d like to show you the basic usage of rsync – a wonderful, old and reliable tool for incremental data transfers and synchronization of local directories or even data between different Unix systems.
rsync is quite a complicated command, so don’t expect this first post to explain everything and cover every possibility. Like I said, this is only the beginning.
What is rsync?
rsync (stands for remote synchronization) is an open source tool for data transfers between Unix systems.
In simplest form, it’s just a Unix command you run locally to synchronize two directories. But the real power of rsync is when you need to synchronize directories between remote systems. rsync relies on ssh protocol for transferring the data between Unix systems, but earlier versions used rsh. Advanced deployments imply using rsync server in addition to simply running the command – this is basically the same command but running in a stand-by daemon mode.
rsync can easily be found or installed in any modern Unix-like OS, but it’s always best to check the official website for latest developments around this tool: rsync website.
What does rsync do?
rsync synchronizes directories – makes one directory look (contain the same files and subdirectories) exactly like another one. rsync works by getting a list of files in your source and destination directories, comparing them as per specified criteria (file size, creation/modification date or checksum) and then making the destination directory reflect all the changes which happened to the source since the last synchronization session.
Basic rsync usage
Just to show you how it works, I’m going to create two directories with a few files in them. /tmp/dir1 in my examples will be a source directory (original dataset), while /tmp/dir2 will be a destination directory – to be made the same as /tmp/dir1 as the result of running rsync.
So that’s how I set up directories and files:
ubuntu$ mkdir /tmp/dir1 /tmp/dir2 ubuntu$ cd /tmp ubuntu$ echo "original file 1" > dir1/file1 ubuntu$ echo "original file 2" > dir1/file2 ubuntu$ echo "original file 3" > dir1/file3 ubuntu$ cp dir1/file1 dir2
That’s how our directories and files look now, so dir2 contains a copy of file1:
ubuntu$ find ./dir* ./dir1 ./dir1/file2 ./dir1/file3 ./dir1/file1 ./dir2 ./dir2/file1
Now it’s time to run your first ever rsync. There’s two ways of specifying options for the command, a full option name starting with — and usually having a meaningful name, or a short option name – starting with – and having short meaningless names (usually one-letter ones) for each option.
The last two parameters in an rsync command line should be the source and the destination directories.
In this example below, we’re using the following options:
-avz – a for archive mode (preserve all the attributes of each file and directory – ownership, permissions, etc), v for verbose mode (report a list of files processed by rsync) and z for data compression to speed transfers up.
–stats – this option shows a summary at the end of rsync’ing process to highlight the main stats of the job
ubuntu$ rsync -avz --stats /tmp/dir1/ /tmp/dir2 building file list ... done file2 file3 Number of files: 4 Number of files transferred: 2 Total file size: 48 bytes Total transferred file size: 32 bytes Literal data: 32 bytes Matched data: 0 bytes File list size: 87 File list generation time: 0.001 seconds File list transfer time: 0.000 seconds Total bytes sent: 221 Total bytes received: 64 sent 221 bytes received 64 bytes 570.00 bytes/sec total size is 48 speedup is 0.17
Stats are self-explanatory, and you can see that although there were 4 files found in source directory /tmp/dir1, only 2 files were transferred into /tmp/dir2 because /tmp/dir2 already had one of the files.
That’s all I have for you today, in the next post on rsync I’ll show you some more advanced uses of this command. For the time being, read man rsync or even rsync –help on your system to get an idea of how really powerful this tool is.
Until next time – good luck with your Unix experiments!
If you’re interested in learning more, you should consider buying the following books: