Most useful options in rsync

The rsync command line tool is an invaluable tool for advanced copying and syncing operations, particularly useful for efficiently making and maintaining backups. It can be used to copy and sync files from any source and to any destination both locally and remotely. It is famous for its ability to copy only the differences in files rather than the whole files when they have been modified, which drastically reduces the required bandwidth.

Rsync commands are written in the following format: rsync -options [source] [destination].

Here is an example:

rsync -avz data/ /media/backups/data

This will copy all the files and folders from the data directory to /media/backups/data, which could be on your external backup device. This example already contains three commonly used and useful options:

-a copies recursively (all directories and subdirectories) while preserving symbolic links, permissions, file ownerships and timestamps. Similar to -r except that -r does not preserve permissions and timestamps-

-v shows verbose output, telling you more of what it's doing.

-z uses compression when transferring, which can speed up the transfer, especially over the network.

There are, of course, other useful options that can be added, such as:

-h displays numbers in a human readable form, easier to understand the amount of data transferred.

–stats shows statistics on file transfers if you want even more insight into the statistics of the transferred data.

-m prunes empty directories

-n so called "dry run", that is, it simulates what it would do with a given command without actually making any changes. Could be very useful if you want to make sure you don't screw up and end up losing your data.

-p preserve permissions, in case you're using the -r option instead of -a or don't use recursive syncing, but want to preserve permissions.

-e allows specifying the remote shell to use when you want to use the remote source or destination.

With the -e option you can, for example, use ssh to transfer data with rsync. It would look something like this:

rsync -avze ssh data user@example.com:/home/backups/

When you run a command like this it will ask you for your remote SSH server password, and then copy over the contents of the data directory to /home/backups/ on the remote system.

As with most standard Linux and UNIX utilities, you can run man rsync to quickly get more information about what other options are available, but this should cover most important uses.

5 things you can do with netstat command

The netstat command, which stands for "network statistics", can show you a lot of information about your network including statistics on connections to and from others on the network, used network interfaces, services, ports, and routing tables.

So what could all this information be used for? Just running netstat alone will give you an overview of your network, which will show a list of addresses connected to your system, over which port they're connected, and what services or programs they're talking to.

Here are five relatively simple examples of what you can actually do with netstat.

1. Show who is connected to your system

One of the most useful things you can do with netstat is show exactly who is connected to your system either through an incoming or outgoing connection (whether it is your system which initiated it or the other system). This will simply list all of them:

netstat -a

Look at the "Foreign Address" column to see where the connection is coming from, and "Local Address" to see what on the local machine is it connected.

The following command will show just the TCP (-t) and UDP (-u) connections:

netstat -tua

If you want to turn off hostnames, or domain names, and display only IP numbers just add the -n option.

netstat -tuan

If you want it to display this continuously to see as connections come and go add the -c option.

netstat -tuanc

Needless to say, perhaps, with IP addresses of everyone connecting revealed you can use other tools like traceroute to determine where exactly is it coming from.

2. Show listening ports

If you'd like to see which services are actually listening for incoming connections, perhaps to ensure you don't have something listening that you don't want to be listening, just use the -l option.

netstat -l

You can also limit this to only a specific type of traffic, like TCP in this example (for UDP just use -u):

netstat -lt

3. Find the port used by a program

We can get a little bit more specific by combining the netstat command with other common UNIX utilities like grep, in this example, where we make it easier to find which port is used by a program. We use grep to conveniently dig this info out of the netstat output:

netstat -ap | grep znc

In this example we get a list of all connections mentioning ZNC with the ports it is using, and addresses it is connected to.

4. Check on the routing table

With netstat you can easily see the kernel IP routing table being used on your system using the -r option:

netstat -r

5. Show all statistics

Being a statistics utility you can of course see a summary of a great number of statistics about your system's networking. Just run the netstat command with the -s option:

netstat -s

This will display a huge list of statistics, but you'll immediately recognize the most interesting ones depending on what you're looking for. For example you can see a total number of packets received, number of active TCP connections, and a number of extended more detailed statistics for each protocol.

Note

These examples are based on netstat in Linux, where it has been succeeded by the ss command from the iproute2 package, but it should apply to most UNIX and UNIX like systems. You can also check the manual page readily available via the man netstat command for more information.

How to get started with tmux

Quite simply, tmux allows you to run multiple terminal command lines at the same time side by side much like putting multiple windows next to each other so you see both simultaneously. It stands for "terminal multiplexer", a fancy term for something so logical. Here are the basics to get you started with tmux quickly.

First of all, just run tmux. You will very likely have it in your system no matter which Linux or UNIX distribution you're using. Otherwise you can get it from the tmux website.

tmux

And you're in. Not much has changed, but you should see the green bar at the bottom, which is the tmux status bar showing you which window you're in, date and time, and the hostname of your system. The goodness is in the keyboard commands that you can now use to split the current window into multiple panes, or to create new windows with their own panes. You can then switch between panes and windows with ease.

The way keyboard commands work is by first pressing a shortcut CTRL-B, and then entering a character corresponding to a command. For example, to split the current window or pane into two vertical panes press CTRL-B %. Just press CTRL-B together, then enter the % normally, and it will split.

Here are a few of the basic commands that will have you up and going with tmux with ease:

  • CTRL-B % – split into two vertical panes (as mentioned)
  • CTRL-B " – split into two horizontal panes
  • CTRL-B z – make the current pane full screen (press again to exit full screen)
  • CTRL-B arrow-key – switch between panes with arrow keys. For example
  • CTRL-B up-arrow – will switch to the pane above the currently active one.
  • CTRL-B c – create a new window. You will notice it indicated in the green status bar under a number.
  • CTRL-B number – switch to a window. To switch to a window 0, for example, press CTRL-B 0.

With these shortcuts you can create as many panes and windows as you need, and switch between them.

One more thing though. These sets of windows and panes are known as tmux sessions, and you can actually have multiple of them, and they are saved on the system so that they can be attached to different terminals you can in from.

You can detach from the current session with CTRL-B d, and you'll end up back on the normal bare command line. Then you can run this to list all of the sessions with its numbers:

tmux -l

You'll see something like:

0: 2 windows (created Tue Sep 2 18:30:02 2014) [237×55]
1: 1 windows (created Tue Sep 2 19:04:03 2014) [237×55]

Then you can attach to session 0, for example, with this command:

tmux attach -t 0

And if you no longer want a particular session you can kill it:

tmux kill-session -t 0

You can see the full breakdown of sessions, their windows, and their panes by issuing this command:

tmux info

Other useful commands with more information:

tmux list-keys
tmux list-commands
man tmux

How to update grub boot loader config

GRUB boot loader starts up what's necessary for your Linux or UNIX system to boot up. You can edit its settings, like various boot options and which operating systems to select from, by editing the the /boot/grub/grub.cfg or /etc/grub.conf depending on your system. Graphical programs are also available for this purpose. See our GRUB Boot Loader overview for more.

Once you've edited your configuration you'll need to update grub to use it. This is very easily done by this single command:

sudo update-grub

Then once you reboot your new config should be active.

How to change filesystem label with tune2fs

Some properties of ext2, ext3, and ext4 file systems on Linux and UNIX can be tuned on the fly using the tune2fs command. This includes the file system's label.

First of all let's list the existing values of a given file system using the -l option:

tune2fs -l /dev/sda1

You can also use dumpe2fs /dev/sda1 to list a lot more of the information about the file system, but the above command will neatly list all of the tunable values including the "Filesystem volume name", which is the file system label.

To change the label use the -L or –volume-label option followed by the new desired label. Keep in mind ext2 file system labels can be only 16 characters long, and will otherwise be truncated.

tune2fs -L /dev/sda1 MyFilesystem

Of course, replace "MyFilesystem" with your own desired label and /dev/sda1 with your own device. After you set the label you can specify this file system by its label when using programs like fsck and mount or in the /etc/fstab configuration file by using LABEL=MyFilesystem.

Keep iptables rules after reboot

The iptables command on Linux allows setting the rules for the Linux built-in firewall to follow when filtering packets flowing through the system. The iptables command applies to IPv4 packets and the ip6tables applies to IPv6 packets. When you make modifications to your set up you can save them using the iptables-save command for IPv4 rules and ip6tables-save for IPv6 rules:

In Debian or Ubuntu systems you would therefore do this for IPv4:

iptables-save > /etc/iptables/rules.v4

And this for IPv6:

ip6tables-save > /etc/iptables/rules.v6

And the same for RedHat Enterprise Linux or CentOS:

iptables-save > /etc/sysconfig/iptables
ip6tables-save > /etc/sysconfig/ip6tables

Then you would use the iptables-restore command to restore the saved rules:

iptables-restore < /etc/iptables/rules.v4

Manually restoring your own rules every time you boot the system may be a chore. Luckily there is an easy way to do this automatically. On Debian or Ubuntu just use the iptables-persistent package:

apt-get install iptables-persistent

If you saved your rules in /etc/iptables/rules.v4 as specified above they will load automatically on every boot.

For RHEL or CentOS systems you can simply enable the iptables service:

chkconfig iptables on

And make sure your rules are saved:

service iptables save

How to capture network traffic with tcpdump

With tcpdump you can intercept, read, and save TCP/IP packets flowing through a particular network interface. These packets, which are the fundamental unit of data being transmitted over a TCP/IP network such as the internet, consist of two kinds of data. One is control data and the other is user data. Control data is the information about where the user data is to be delivered, where it's coming from, what is its length, and other information about the actual user data. The user data is the actual data being transmitted, which could include just about anything. It could even include passwords and usernames if this data is sent in clear text and not encrypted.

Simply running tcpdump on the command line will capture and display packets flowing through the eth0 network interface, which is the typical default interface used. However, it will only be indiscriminately listing packets with their control data, and you wont actually see any user data. To display that you'll need to run tcpdump with the -X option:

tcpdump -X

To make what you're getting more useful though we can use a few options. For example, we could save this stuff in a file instead of having it just be dumped on our screen, which makes it pretty hard to read anyway:

tcpdump -X -w packets.txt

Once you run this your packets.txt file will start getting filled up with lots of information really quickly so long as there's any traffic flowing through eth0. Let's say that you're running a web server and someone visits your web site. You would see the HTML contents of the web page being requested in the packets.txt file as user data of that packet. You see everything that's being transmitted. If what is being transmitted is by any chance encrypted though you might only see incomprehensible gibberish, but not making it easy to discern what's being transmitted by intercepting these packets is the whole point of encryption.

What if you wanted to read another network interface, like eth1? Simple, just tell it to capture eth1 packets with the -i option:

tcpdump -X -w packets.txt -i eth1

To listen for any and all traffic, just use -i any instead, and it will listen to all network interfaces.

Here are a few more useful options that help you specify what you want to capture and have dumped by tcpdump. To see all of the options you can check the manpage by running man tcpdump.

To disable resolving hostnames and domains, which can save a bit of time, and display only IP addresses use the -n option. To disable port names, use -nn. With these options the first example would look like this:

tcpdump -Xnn

To show only a certain number of packets and then stop instead of running indefinitely you can specify the -c 20 option, where -c stands for "count", and "20" would represent 20 packets.

tcpdump -Xnnc 20

Finally, if you want to make absolutely sure you see the maximum possible information that is being captured use the verbosity options. You can increase verbosity up to three times. With just -v, -vv, or -vvv for maximum verbosity. Also, we can use the -S option to show absolute rather than relative sequence numbers just to make sure we see the actual numbers. So let's construct a command that would show the maximum possible information on a sample of 100 packets, and store it into packets.txt.

tcpdump -XSvvvc 100 -w packets.txt

And that should get you on the right track to playing with and learning network traffic capture with tcpdump.

How to use dpkg to compare two Linux servers

While we mainly install, remove, update and otherwise manage software on Debian and Ubuntu based systems using apt-get, the lower level packaging system that apt-get actually relies on is dpkg.

You can use dpkg to list all of the installed packages on the current system. Do this by passing the –get-selections option. The following is the command that would get all of the packages, sort them, and list them into an installed-packages file.

dpkg --get-selections|sort > installed-packages

Now you can do the same on your second Debian or Ubuntu server except you might want to name your file something like installed-packages2.

With those two files ready you can now compare them. Copy over the first file to the second system (or vice versa) and run the diff command to see the differences.

diff -u installed-packages installed-packages2 > compare-servers

This would compare the two lists and throw the results into compare-servers text file. Name it whatever you like. The -u option makes the results a little more readable, but you can alternatively use the -y option which will format the results in two columns, first representing the first file, and second representing the second file.

Studying the differences between package lists can help you figure out what the differences are between two server setups in terms of installed software, what does one have that the other doesn't. If you want to replicate installed packages of one system to another, or in other words, install all of the packages which are installed on the first system to the second system, you can do that with dpkg and apt-get.

First run the following to get dpkg to select packages to install, marking them for installation:

dpkg --set-selections < installed-packages 

And then run this apt-get command to get those packages installed:

apt-get dselect-upgrade

This also allows you to quickly and easily restore a given system after a fresh install, if you've saved the list of installed packages from a previous system before you deleted it. It is also possible to compare and restore system settings from the /etc directory, but that is beyond the scope of this post.

How to install unrar in linux

The unrar program, which serves to open and extract popular .rar archives, is often available for install from repositories of a given Linux distribution. That should make installing it easy by using your distribution's package management system. That can be either a graphical user interface program like Ubuntu Software Center, or a command like tool like apt-get.

Some distributions may, however, require you to enable or add an additional repository to those included by default, which is usually the one containing various proprietary packages. This is because unrar, with the exception of the unrar-free package (which doesn't support all .rar files), is proprietary software. It's not open source. Some distributions avoid including non-free or non open source software by default, because they want to encourage using only Free Open Source Software, either for philosophical or practical reasons.

With that said, here is how to install unrar in the few most popular Linux distributions.

Ubuntu, Linux Mint

This also covers all of the Ubuntu variants like Xubuntu, Kubuntu, and Lubuntu. I know, those names are kinda hilarious when you string them together like that! But on to the install:

sudo apt-get install unrar

Debian

Debian is the grandaddy of Ubuntu, but it follows a quite different philosophy, and so does not enable the non-free repository by default. To enable it run (or copy-paste) the following command:

sudo echo 'deb ftp://ftp.us.debian.org/debian/ wheezy non-free
deb http://security.debian.org/ wheezy/updates non-free deb http://volatile.debian.org/debian-volatile wheezy/volatile non-free' >> /etc/apt/sources.list.d/wheezy.non-free.list

If by the time you read this you're running a newer version than Debian 7 (codenamed "wheezy"), just replace "wheezy" above with the new codename.

After adding the repository you should update the package database to make new packages available for install:

sudo apt-get update

And then you can install the same way you would in Ubuntu:

sudo apt-get install unrar  

Fedora

In Fedora you need to add a RPM Fusion Non-Free repository before you can install unrar. Not to worry, this is pretty easy. Just follow the simple instructions provided at RPMFusion.org.
It involves downloading and launching a couple of files, and following prompts on the screen. Command line set up options are also shown.

Once you have it you can simply run the following command to install:

sudo yum install unrar

Speaking of Fedora it may be worth mentioning that Korora, a Fedora-based distribution, enables this repository by default so if you use Korora all you need to do is run the above yum command.

openSUSE

sudo zypper install unrar

That should be it for openSUSE since the "non-oss" (non open source) repository, which contains unrar, is added and enabled by default.

Arch Linux

pacman -S unrar

Same story as openSUSE.

Other

If you're running any other Linux distribution chances are it is a derivative of any of the above or otherwise contains unrar in its official repositories. As a last resort option, which you probably wont need, you can download unrar directly from the RarLabs web site.

The RAR for Linux package, available for download there, contains both unrar and rar binaries as well as the makefile that allows you to easily install them. Just extract the package to any directory, then in the command line change the directory you are in to the extracted directory, and run the make command as a superuser.

In other words:

cd Downloads/rar/ && sudo make

The make command will copy the binaries to locations where Linux is looking for binaries, so you can run the rar and unrar commands as normal.

Don't do this, however, unless you really have to. Chances are your distro has unrar packaged up already and available for install. You can also check for that at pkgs.org (if your distro is listed).

Most popular BSD distributions

BSD stands for Berkeley Software Distribution, and it was a UNIX software compilation released by the Computer Software Research Group at the University of California, Berkeley between 1977 and 1995. The first one was 1BSD (First Berkeley Software Distribution) compiled by Bill Joy, but only as an add-on to Sixth Edition Unix from Bell Labs. In 2BSD Bill Joy added a C Shell and the iconic vi text editor.

As BSD evolved it became a complete UNIX operating system in its own right. The last release from Berkeley was 4.4BSD-Lite Release 2, and it contained no proprietary code from AT&T, and was freely available under the permissive BSD license. Since then other projects, descendants of the original BSD, continued the development and they are what BSD today generally refers to. Most popular of these are FreeBSD, NetBSD, and OpenBSD.

BSD was the first to include support for the Internet Protocol stack in form of its Berkeley sockets, which made it easy to read and write files over the network. It is also easier to natively run software from other operating systems on BSD thanks to its binary compatibility layer. The very permissive nature of the BSD open source license enabled widespread use of its code in various other software projects. Apple's OSX and iOS, for example, are based on BSD code.

FreeBSD

Originally based on top of 386BSD, and since version 2.0 on 4.4BSD-Lite FreeBSD exists with a goal of providing a complete operating system that can serve any purpose without any strings attached, free in every sense of the word. FreeBSD originated the ports collection system for easy download, building, and installation of software packages that continues to be one of the easiest and most sophisticated ways to install software in the UNIX world. This was adopted by NetBSD and OpenBSD as well.

FreeBSD also uses a rather open model of development by letting hundreds of "committers" make changes to FreeBSD at any time as needed. The selection of committers and resolution of any disputes is managed by the elected Core Team.

FreeBSD is the most popular BSD version used, and also served as the basis of many other operating systems such as most notably the Apple's OSX. There is also a number of FreeBSD variants such as the desktop oriented PC-BSD created to be easy for everyone to use.

NetBSD

NetBSD was founded after FreeBSD with major emphasis on portability at a time when FreeBSD was mostly focusing on the x86 architecture. NetBSD runs on so many platforms that they have a slogan saying "Of course it runs BSD". This makes it particularly suitable for computer research because it readily runs on both old and new architectures alike. NetBSD also uses the pkgsrc package management system originally based on the Ports collection system from FreeBSD.

OpenBSD

In 1995 Theo De Radt forked the NetBSD project to create OpenBSD. Particular focus was put on security as well as strong emphasis on great documentation, code correctness, and open source licensing. The project has also spawned a number of key widely used security tools like OpenSSH, OpenNTPD, PF, and most recently the LibreSSL fork of OpenSSL after the Heartbleed bug fiasco. Unlike FreeBSD the project is more strictly managed by Theo De Radt himself. OpenBSD also supports about 20 different hardware architectures.