How To Synchronize Directories with Rsync

Today I’d like to show you the basic usage of rsync – a wonderful, old and reliable tool for incremental data transfers and synchronization of local directories or even data between different Unix systems.

rsync is quite a complicated command, so don’t expect this first post to explain everything and cover every possibility. Like I said, this is only the beginning.

What is rsync?

rsync (stands for remote synchronization) is an open source tool for data transfers between Unix systems.

In simplest form, it’s just a Unix command you run locally to synchronize two directories. But the real power of rsync is when you need to synchronize directories between remote systems. rsync relies on ssh protocol for transferring the data between Unix systems, but earlier versions used rsh. Advanced deployments imply using rsync server in addition to simply running the command – this is basically the same command but running in a stand-by daemon mode.

rsync can easily be found or installed in any modern Unix-like OS, but it's always best to check the official website for latest developments around this tool: rsync website.

What does rsync do?

rsync synchronizes directories – makes one directory look (contain the same files and subdirectories) exactly like another one. rsync works by getting a list of files in your source and destination directories, comparing them as per specified criteria (file size, creation/modification date or checksum) and then making the destination directory reflect all the changes which happened to the source since the last synchronization session.

Basic rsync usage

Just to show you how it works, I’m going to create two directories with a few files in them. /tmp/dir1 in my examples will be a source directory (original dataset), while /tmp/dir2 will be a destination directory – to be made the same as /tmp/dir1 as the result of running rsync.

So that's how I set up directories and files:

ubuntu$ mkdir /tmp/dir1 /tmp/dir2
ubuntu$ cd /tmp
ubuntu$ echo "original file 1" > dir1/file1
ubuntu$ echo "original file 2" > dir1/file2
ubuntu$ echo "original file 3" > dir1/file3
ubuntu$ cp dir1/file1 dir2

That's how our directories and files look now, so dir2 contains a copy of file1:

ubuntu$ find ./dir*
./dir1
./dir1/file2
./dir1/file3
./dir1/file1
./dir2
./dir2/file1

Now it's time to run your first ever rsync. There's two ways of specifying options for the command, a full option name starting with — and usually having a meaningful name, or a short option name – starting with – and having short meaningless names (usually one-letter ones) for each option.

The last two parameters in an rsync command line should be the source and the destination directories.

In this example below, we're using the following options:

-avz – a for archive mode (preserve all the attributes of each file and directory – ownership, permissions, etc), v for verbose mode (report a list of files processed by rsync) and z for data compression to speed transfers up.

–stats – this option shows a summary at the end of rsync'ing process to highlight the main stats of the job

ubuntu$ rsync -avz --stats  /tmp/dir1/ /tmp/dir2
building file list ... done
file2
file3

Number of files: 4
Number of files transferred: 2
Total file size: 48 bytes
Total transferred file size: 32 bytes
Literal data: 32 bytes
Matched data: 0 bytes
File list size: 87
File list generation time: 0.001 seconds
File list transfer time: 0.000 seconds
Total bytes sent: 221
Total bytes received: 64

sent 221 bytes  received 64 bytes  570.00 bytes/sec
total size is 48  speedup is 0.17

Stats are self-explanatory, and you can see that although there were 4 files found in source directory /tmp/dir1, only 2 files were transferred into /tmp/dir2 because /tmp/dir2 already had one of the files.

That’s all I have for you today, in the next post on rsync I’ll show you some more advanced uses of this command. For the time being, read man rsync or even rsync –help on your system to get an idea of how really powerful this tool is.

Until next time – good luck with your Unix experiments!

Related books

If you're interested in learning more, you should consider buying the following books:


Backup & Recovery book

Backup & Recovery

See also:

Share and Enjoy

  • Facebook
  • Twitter
  • Delicious
  • LinkedIn
  • StumbleUpon
  • Add to favorites
  • Email
  • RSS
  • Mark

    I am not sure I understand this, how does this differer say from GoodSync? I've used GoodSync and it has been great but is this something I have to download to get it to work, or are you saying I need to know programming?

  • Gleb Reys

    Hi Mark,

    Thanks for asking! Since I've never used GoodSync, I can't say how exactly rsync is different, apart from the following:

    – it's open source, which means it can be compiled and used on any OS. GoodSync appears to be a Windows application
    – rsync is a well documented tool with protocol open for comments and improvements. GoodSync seems to be a proprietary software
    – GoodSync has graphics interface, while rsync itself is a command line tool. There are quite a few frontends for it, but they need to be downloaded separately

    Bottom line is: you don't need to know programming, but command line-driven rsync will seem harder to use than GoodSync, at least initially

  • Pingback: Tracking the Progress of Rsync Transfers | UNIX Tutorial: Learn UNIX()

  • http://bsodmike.com Michael

    Nice tutorial and example as well. Never had the need to use rsync before; was more curious as to its "power".

    Cheers
    Mike.

  • http://techbyct.com/node/1 Brent2

    I'm somewhat familiar with rsync and have used prebuilt commands in the past. This morning I got inspired and started reading everything I could find.

    The annoying thing? Everything I found was either way too advanced or way too patronizing. This was perfect. I fully intend to see if I can find your next articles on it.

  • tetsuo

    Very useful article. Many thanks! :)

  • tetsuo

    My unix skills are limited and I had to find a simple backup solution for some directories on a server at work. I am now using rsync to sync them with an external drive. Have a great day!

  • obaino

    Thank you for this tutorial.
    I was wondering…
    How is it possible to simultaneously synchronize 2 directories? If there are changes to both directories to keep only the newer files

    thx once again :-)

  • http://microsoft.com bill

    hi all,

    here is what i'm doing :

    # rsync -avuz source/ dest/
    # rsync -avuz dest/ source/

    in order to sync 2 different directories

    if you know what you are doing you can also use the –delete option, in order to remove files that are in "dest" dir and not in "source" dir anymore.

  • James