Tracking the Progress of Rsync Transfers

In my first introductory rsync post, How To Synchronize Directories with Rsync, I've shown you the most basic approach to syncing two directories up. Today, I'd like to show you another useful thing you can do with rsync.

Just to remind you all, rsync is a remote synchronization tool. This means that its primary use lies in the filed of synchronizing files and directories between remote Unix systems, but I feel that it's really important for you to understand the basics before moving on to more advanced stuff.

The beauty of rsync is that its syntax and command line options are exactly the same for local and remote directories synchronizations, so it's not like you'll have to learn it all from scratch when I show you the real world usage of rsync in my next post.

rsync experiments setup

Just use the following commands to re-create your rsync playground for this post:

ubuntu$ rm -rf /tmp/dir1 /tmp/dir2
ubuntu$ mkdir /tmp/dir1 /tmp/dir2
ubuntu$ cd /tmp
ubuntu$ echo "original file 1" > dir1/file1
ubuntu$ echo "original file 2" > dir1/file2
ubuntu$ echo "original file 3" > dir1/file3
ubuntu$ cp dir1/file1 dir2

It will probably be more convenient for you to just download the same commands in form of a script though, so here's the link:

Tracking rsync progress

The larger the directories you're synchronizing with rsync, the longer it will take for the command to finish. Depending on the scale of your task, it can be minutes, long hours or even days before you get the synchronization complete. With this in mind, the importance of progress tracking with rsync should be obvious to you.

Here's how you make rsync report its progress: just use the –progress command line option (in addition to the command line I introduced you to last time):

ubuntu$ rsync -avz --stats --progress /tmp/dir1/ /tmp/dir2
building file list ...
4 files to consider
          16 100%    0.00kB/s    0:00:00 (xfer#1, to-check=1/4)
          16 100%   15.62kB/s    0:00:00 (xfer#2, to-check=0/4)
Number of files: 4
Number of files transferred: 2
Total file size: 48 bytes
Total transferred file size: 32 bytes
Literal data: 32 bytes
Matched data: 0 bytes
File list size: 59
File list generation time: 0.001 seconds
File list transfer time: 0.000 seconds
Total bytes sent: 193
Total bytes received: 64
sent 193 bytes  received 64 bytes  514.00 bytes/sec
total size is 48  speedup is 0.19

Why you should track the progress of rsyncs

The key differences are that you first get a regularly updated message about building a file list. When you run into hundreds of thousands of files, this becomes quite useful:

building file list ...
4 files to consider

Another way to track progress is to see the transfer progress for each of the involved files. For thousands tiny files it won't be all that impressive a feature, but if you're transferring huge files between remote systems, a dynamic transfer progress for each file will be exactly what you need:

          16 100%    0.00kB/s    0:00:00 (xfer#1, to-check=1/4)
          16 100%   15.62kB/s    0:00:00 (xfer#2, to-check=0/4)

That's it, hope you like this new option. I promise to tell you more some other time!

See also:

  • Thanks For The Information. I am currently playing with rsync on ubuntu-server 8.10 over ssh and came across your site. Great Job!

  • Gleb Reys

    You're welcome, John! ssh is going to be covered in the next post on rsyncs.

  • At the end it mentions:

    speedup is 0.19

    What is that referring to?

  • Rayan

    Hi can u please help me on how to calculate the total timings of the data transfer using the rsync command.
    I tried giving –stats –progress –times -av but it doesnt show the total time utilized for syncing. It shows individual timings which is very hard to total it.please help me

    Thanks for the help in advance

  • Thanks Alot~~~ , I'll look up the avz command parameters

  • Tony D

    speedup is the factor of how much data needed to be synced up between the 2 systems and how much actually transferred, e.g.

    sent 60445311 bytes received 4622 bytes 2811624.79 bytes/sec
    total size is 100958200639 speedup is 1670.11

    100958200639 divided by (60445311 + 4622) is 1670.1126

    Another example:

    sent 44865238312 bytes received 906 bytes 23719396.89 bytes/sec
    total size is 44866713376 speedup is 1.00

    44866713376 divided by (44865238312 + 906) is 1.0000328

    It tells how much data you did not have to transfer to sync them up compared to using a straightforward cp command