Tracking the Progress of Rsync Transfers

In my first introductory rsync post, How To Synchronize Directories with Rsync, I've shown you the most basic approach to syncing two directories up. Today, I'd like to show you another useful thing you can do with rsync.

Just to remind you all, rsync is a remote synchronization tool. This means that its primary use lies in the filed of synchronizing files and directories between remote Unix systems, but I feel that it's really important for you to understand the basics before moving on to more advanced stuff.

The beauty of rsync is that its syntax and command line options are exactly the same for local and remote directories synchronizations, so it's not like you'll have to learn it all from scratch when I show you the real world usage of rsync in my next post.

rsync experiments setup

Just use the following commands to re-create your rsync playground for this post:

ubuntu$ rm -rf /tmp/dir1 /tmp/dir2
ubuntu$ mkdir /tmp/dir1 /tmp/dir2
ubuntu$ cd /tmp
ubuntu$ echo "original file 1" > dir1/file1
ubuntu$ echo "original file 2" > dir1/file2
ubuntu$ echo "original file 3" > dir1/file3
ubuntu$ cp dir1/file1 dir2

It will probably be more convenient for you to just download the same commands in form of a script though, so here's the link: rsync-setup.sh

Tracking rsync progress

The larger the directories you're synchronizing with rsync, the longer it will take for the command to finish. Depending on the scale of your task, it can be minutes, long hours or even days before you get the synchronization complete. With this in mind, the importance of progress tracking with rsync should be obvious to you.

Here's how you make rsync report its progress: just use the –progress command line option (in addition to the command line I introduced you to last time):

ubuntu$ rsync -avz --stats --progress /tmp/dir1/ /tmp/dir2
building file list ...
4 files to consider
file2
          16 100%    0.00kB/s    0:00:00 (xfer#1, to-check=1/4)
file3
          16 100%   15.62kB/s    0:00:00 (xfer#2, to-check=0/4)
Number of files: 4
Number of files transferred: 2
Total file size: 48 bytes
Total transferred file size: 32 bytes
Literal data: 32 bytes
Matched data: 0 bytes
File list size: 59
File list generation time: 0.001 seconds
File list transfer time: 0.000 seconds
Total bytes sent: 193
Total bytes received: 64
sent 193 bytes  received 64 bytes  514.00 bytes/sec
total size is 48  speedup is 0.19

Why you should track the progress of rsyncs

The key differences are that you first get a regularly updated message about building a file list. When you run into hundreds of thousands of files, this becomes quite useful:

building file list ...
4 files to consider

Another way to track progress is to see the transfer progress for each of the involved files. For thousands tiny files it won't be all that impressive a feature, but if you're transferring huge files between remote systems, a dynamic transfer progress for each file will be exactly what you need:

file2
          16 100%    0.00kB/s    0:00:00 (xfer#1, to-check=1/4)
file3
          16 100%   15.62kB/s    0:00:00 (xfer#2, to-check=0/4)

That's it, hope you like this new option. I promise to tell you more some other time!

See also:

Share and Enjoy

  • Facebook
  • Twitter
  • Delicious
  • LinkedIn
  • StumbleUpon
  • Add to favorites
  • Email
  • RSS
  • http://www.variux.com John Anderson

    Thanks For The Information. I am currently playing with rsync on ubuntu-server 8.10 over ssh and came across your site. Great Job!

  • Gleb Reys

    You're welcome, John! ssh is going to be covered in the next post on rsyncs.

  • http://techbyct.com/node/1 Brent2

    At the end it mentions:

    speedup is 0.19

    What is that referring to?

  • Rayan

    Hi can u please help me on how to calculate the total timings of the data transfer using the rsync command.
    I tried giving –stats –progress –times -av but it doesnt show the total time utilized for syncing. It shows individual timings which is very hard to total it.please help me

    Thanks for the help in advance

  • http://blog.littlebearz.com littlebear

    Thanks Alot~~~ , I'll look up the avz command parameters