While Mac OS is unlike any Unix-like operating system I've managed so far, there are certainly some of similarities. I can honestly say that I'm enjoying the Mac Book Pro so far, and hope to discover most of the differences compared to my previous Unix-like desktop which is Ubuntu 9.10.

Mounting NFS on MAC OS X

One thing which I noticed immediately was that out of the box it was impossible to mount any NFS shares from my Ubuntu NAS server. Any attempt to mount a remote filesystem would give me an error like this:

mbp:~ root# mount nasbox:/try /mnt
mount_nfs: /mnt: Operation not permitted

This error was happening for a relatively simple NFS share on the Ubuntu box:

nasbox# cat /etc/exports
/try            (rw)

… so I started looking around and realized that the reason for this strange problem is quite simple.

Mac OS X uses non-standard port for outgoing NFS connections

That's right – apparently, Mac OS X uses a non-privileged port (2049) for TCP and UDP connections serving the NFS transport. What this means is that most likely attempts to mount remote filesystems will fail, because most NFS servers don't really like connections from insecure ports.

There are two ways to approach this problem:

1. Fix it on the client side (probably makes more sense)
2. Fix it on the NFS server side (if you manage both systems)

Using reserved NFS port number on Mac OS X

There's a mount option supported by the mount_nfs command, which allows you to force the NFS client connections to originate from a privileged port. This will magically make your attempts to mount remote filesystems successful. The option is called resvport.

First we double-check that default mounts still don't work:

mbp:~ root# mount nasbox:/try /mnt
mount_nfs: /mnt: Operation not permitted

… and now let's use the resvport option:

mbp:~ root# mount -o resvport nasbox:/try /mnt

… and make sure we're actually looking at a mounted filesystem:

mbp:~ root# df -h /mnt
Filesystem   Size   Used  Avail Capacity  Mounted on
nasbox:/try    61Gi   56Gi  2.6Gi    96%    /mnt

Allowing connections from non-privileged ports on NFS server

Like I said, if you manage both the Mac OS based client and the NFS server, perhaps it makes more sense to relax the default NFS server security and allow the connections from non-privileged ports.

Just to remind you about the validity of such a decision, the option to allow non-privileged connections is called insecure.

Here's how you use it:

nasbox# cat /etc/exports
/try            (rw,insecure)

After making this change to the /etc/exports file, you'll have to restart your NFS server. On my Ubuntu NAS box, it's done like this:

nasbox# /etc/init.d/nfs-kernel-server restart
* Stopping NFS kernel daemon
…done.
* Unexporting directories for NFS kernel daemon…
…done.
* Exporting directories for NFS kernel daemon…
…done.
* Starting NFS kernel daemon
…done.

We know are ready to attempt the default mount of the same filesystem on the Mac OS X client:

mbp:~ root# mount nasbox:/try /mnt

That's it! I won't promise any Mac OS posts just yet, but if there is enough interest – I'd love to do the research and to post all the discoveries on the Unix Tutorial pages.

I'm back from my holidays, and it's time to open the Unix Tutorial members area for new registrations once again!

If you're still looking for a reason to become a member – please read the Becoming a Unix Tutorial member post.

This time around, new users will receive a life-time access to just one course, the Unix foundations one. This is going to be a constantly updated and expanded course so I'm sure you'll benefit from revisiting it now and then. New courses added at a later stage will never be available for free: I plan to make them very specialized and will charge for access – more on this later.

Wait no more – here's your chance to join: Unix Tutorial members sign-up. I don't have nearly as much time to further develop the course, but please take time to leave a comment to this post or update a thread on the Unix Tutorial members discussion board about what exactly you're interested in.

Basic calculations with bc

bc is a very simple command. It takes standard input as an expression and then evaluates this, performing all the necessary calculations and showing you the result. Thus, to quickly sum numbers up or get a result of some other calculation, simply echo the expression and then pipe it out to the bc command:

ubuntu$ echo "1+2" | bc
3

Now, in scripts your calculations with bc are done quite similarly to what we did in bash. Here's an example:

ubuntu$ NUMBER1=1
ubuntu$ NUMBER2=2
ubuntu$ SUM=$(echo "$NUMBER1+$NUMBER2"| bc)
ubuntu$ echo $SUM
3

I told you these calculations would be basic, right? Now onto the more interesting stuff!

Fixed point calculations with bc

Most people learn about bash math limitations when they attempt to do a simple calculation but can't get the current answer with fixed point values. By default, all the operations happen with integers, and that's what you would get:

ubuntu$ echo "1/2" | bc
0

Now, if you expect 0.5 to be the result of dividing 1 by 2, you need to explain it to bc, because by default it doesn't show you any fractional part of the number.

The way you do this is quite simple: all you have to do is specify the number of digits you'd like to see  after the radix point of your result. For example, if I set this number to 5, I'll get bc to output the result of my calculation with 5 digits after the radix point. The special keyword to convey this intention to the bc command is called scale. Just specify the scale value and separate it from the rest of your expression by the semicolon sign:

ubuntu$ echo "scale=5; 1/2" | bc
.50000

Here's another example:

ubuntu$ echo "scale=5; 0.16*10.79" | bc
1.7264

Hope this answers your question! bc command is very powerful, so I'll definitely have to revisit it again in the future. For now though, enjoy the fixed point calculations and be sure to ask questions if you think I can help!

See also:

• Summing numbers up in Unix scripts
• Basic math in Unix scripting
• Another way of doing calculations in scripts
  

An example of a command alias in Unix shell

Here's one of the most useful aliases I have for Solaris systems:

solaris$ alias ls='/usr/local/gnu/bin/ls --color -F'

What is allows me to do is to simply type "ls" instead of the really long command line it refers to: /usr/local/gnu/bin/ls –color -F.

You see, the ls command which is shipped with Solaris, doesn't have many options of the more up-to-date GNU ls command, and working with many Linux systems I quite like some of them like color highlighting of different directory objects – files, directories and executables.

Try typing the longer command a few times in a row, and compare it to the "ls" to get the idea of how much of a productivity gain one command alias can be!

Why you should use command aliases

Like many other things in Unix, aliases are a way to become more productive. The general rule of thumb is this: if you have to run some command something more than once every day – consider creating an alias for it. These are just a few cases where it makes sense to employ them:

• if you repeatedly check whether some files exist or get updated
• if you're monitoring a certain aspect of your OS and you get the values using grep command
• if you're connecting to the same hosts using rsh or ssh

All of these and many more examples are greatly simplified if you alias them to some shorter commands.

Creating new aliases in bash

Setting up a new alias is quite easy, the syntax for alias command is very straightforward. Let's say I want to automate the confirmation of swap usage based on a free command in Linux:

redhat$ free
total       used       free     shared    buffers     cached
Mem:       2075156     945712    1129444          0     177292     503416
-/+ buffers/cache:     265004    1810152
Swap:      2040244          0    2040244

The result I'm after is this command:

redhat$ free | grep Swap
Swap:      2040244          0    2040244

And here's how I can create an alias called "swp" which refers to this series of commands:

redhat$ alias swp='free | grep Swap'

Once you execute this command, you can start using swp as a Unix command:

redhat$ swp
Swap:      2040244          0    2040244

Important: such a creation of new aliases is going to be only active for your current Unix shell and sub-shells you may spawn. To make your alias permanent, you'll have to update one of your initialization scripts like. For Linux and bash, you should add the same alias command to your .bashrc file.

Removing aliases in Unix

In very much the same way, you can use the unalias command to get rid of a certain alias. The nature of this command is such that you'll most likely use it when creating and debugging new aliases. It's unlikely that you'll need to use it in your initialization scripts.

Following the example above, here's how to get rid of the swp alias and verify that it's gone:

redhat$ unalias swp
redhat$ swp
bash: swp: command not found

How to list your current aliases

If you run the alias command without any parameters, you'll be shown a full list of aliases currently configured for your user account, here's an example from one of my systems:

l.      ls -d .* --color=tty
ll      ls -l --color=tty
ls      ls --color=tty
vi      vim

That's it for today! Stay tuned for a follow-up post which will share some of the examples for command aliases in Unix. If you have some – please leave a comment so that I can share it with others!

See also:

• Unix commands: ls
• List directories in a given directory
• How to find what symlink points to
  

Basic construction for summing up in shell scripts

In my Basic arithmetic operations in Unix shell post last year, I've shown you how to sum up two numbers:

SUM=$(($NUMBER1 + $NUMBER2)

using the same approach, it's possible to calculate sums of as many numbers as you like, if you use one of the loops available in your shell.

Before we get started, let's create a simple file with numbers we'll work with:

ubuntu$ for i in 1 2 3 4 5 6 7 8 9; do echo $i >> /tmp/nums; done;
ubuntu$ cat /tmp/nums
1
2
3
4
5
6
7
8
9

Using a while loop to sum numbers up in Unix

Here's an example of how you can use a while loop in Unix shell for summing numbers up. Save it as a /tmp/sum.sh script, and don't forget to do chmod a+x /tmp/sum.sh so that you can run it!

#!/bin/sh
#
SUM=0
#
while read NUM
do
        echo "SUM: $SUM";
        SUM=$(($SUM + $NUM));
        echo "+ $NUM: $SUM";
done < /tmp/nums

Here's how the output will look when you run it:

ubuntu$ /tmp/sum.sh
SUM: 0
+ 1: 1
SUM: 1
+ 2: 3
SUM: 3
+ 3: 6
SUM: 6
+ 4: 10
SUM: 10
+ 5: 15
SUM: 15
+ 6: 21
SUM: 21
+ 7: 28
SUM: 28
+ 8: 36
SUM: 36
+ 9: 45

Of course, your data will most probably will be in a less readable form, so you'll have to do a bit of parsing before you get to sum the numbers up, but the loop will be organized in the same way.

That's it for today, enjoy and feel free to ask questions!

See also:

• Basic arithmetics in scripts
• Variables in Unix shells
• Unix glossary
  

Why would you want to convert timestamps to Unix epoch time?

Unix epoch time is an ever-growing counter which increments every second and shows the number of seconds elapsed since 00:00:00 UTC on January 1, 1970. Such numbers are pretty useless in their raw form for the tasks of confirming current time and date, but they're perfect for measuring between two points in time.

Current epoch time in Perl

First things first. If you want to confirm the current Unix epoch time in Perl, here's how you do it: just use the time() subroutine.

#!/bin/perl
print "Current (epoch) time: " . time() . "\n";

Converting regular date and time into epoch time

It's equally easy to convert time into Unix epoch representation, for this you should use timegm() or localtime() subroutine from the Time::Local module.

timegm() has the following syntax:

$current = timegm($sec,$min,$hours,$day,$month,$year);

Important: While most of the parameters are self-explanatory, it is probably worth reminding that $year should be specified as a number like 109 and not 2009.
Also, the $month is not the number of a month, but a number of months since January. So, 0 will mean January itself, 1 will mean February, and 11 means December.

Here's a fully working example, showing the countdown in seconds until midnight on the New Year's Eve 2009:

#!/usr/bin/perl
#
use Time::Local;
#
$sec=59;
$min=59;
$hours=23;
$day=31;
$month=11;
$year=109;
#
$current = time();
print "Current (epoch) time: $current\n";
#
$newyear = timegm($sec,$min,$hours,$day,$month,$year);
print "New Year's Eve 2009: $newyear\n";
#
print "Only " .($newyear-$current) . " seconds to go\n";

When you save this script as epoch.pl and make the file executable, here's what you'll get if you run it:

# ./epoch.pl
Current (epoch) time: 1234911622
New Year's Eve 2009: 1262303999
Only 27392377 seconds to go

That's it for today! Hope this little post will save you time when tacking next scripting challenge with Perl.

See also:

• Time and date in Unix scripts
• Date calculations with GNU date command
• Glossary of Unix terms
  

File ownership in Unix

Just to give you a quick reminder, I'd like to confirm that every single file in Unix belongs to some user and some group. There simply isn't a way to create a file without assigning ownership. I've briefly touched the topic of confirming file ownership in Unix before, so today I will simply build on that and show you how to change ownership of files.

Here's a setup for today: I have created a temporary directory with a few files and made myself the owner of all the files:

ubuntu$ ls -al /home/greys/example/
total 12
drwxr-xr-x  3 greys admin 4096 Feb  9 03:55 .
drwxr-xr-x 13 greys greys 4096 Feb  9 03:54 ..
drwxr-xr-x  2 greys admin 4096 Feb  9 03:55 dir1
-rw-r--r--  1 greys admin    0 Feb  9 03:54 file1
-rw-r--r--  1 greys admin    0 Feb  9 03:55 file2

As you can see from this listing, the owner (third field in each line) is my username – greys. The next field is a Unix group of each file's owner – admin in my example.

Changing owner of a file in Unix

Changing file ownership means only updating the association between a Unix user and a file, and nothing else. When you're changing the owner of a file, no data contained in a file is changed.

To change the owner of a file, you need to use the chown command (easy enough to remember: CHange OWNer – chown), with the following syntax:

ubuntu$ chown nobody file1

In this command, nobody is the username of the new owner for a list of files. In my example, the only file we'd like to change ownership for is file1.

It is important to realize that you can only change file ownership as a super-user (root). Any regular Unix user cannot change the ownership of any file, and I'd like to explain why.

Indeed, some people are surprised: if I'm the owner of a given file, why can't I change the ownership for it? That's because transferring the ownership will mean some other Unix user will become the owner of the file(s) in question. So changing ownership is like making a decision not only for yourself, but for the new owner of the files.This is only something a super-user – special administrative account in Unix – can do.

The same logic applies to other people not being able to become owners of your files, even if they're willing to assume the new responsibilities of owning files. They cannot revoke your ownership, because each Unix user is only allowed to make decisions and take actions on his/her own behalf.

That's why you will probably see an error like this if you attempt to change ownership of a file as your own regular Unix user:

ubuntu$ id
uid=1000(greys) gid=113(admin) groups=33(www-data),113(admin)
ubuntu$ chown nobody file1
chown: changing ownership of `file1': Operation not permitted

But if we become root:

ubuntu$ sudo -i
[sudo] password for greys:
ubuntu#

… we'll have no problem changing owners for any files:

ubuntu# cd /home/greys/example
ubuntu# chown nobody file1
ubuntu# ls -l file1
-rw-r--r-- 1 nobody admin 0 Feb  9 03:54 file1

Changing owner for multiple files

If you're going to change owner of a few files, this can easily be done using either a full list of files or a mask.

First, here's an example of updating ownership for a specified list of files (and as you can see, directories as well):

ubuntu# chown nobody file2 dir1
ubuntu# ls -al
total 12
drwxr-xr-x  3 greys  admin 4096 Feb  9 03:55 .
drwxr-xr-x 13 greys  greys 4096 Feb  9 03:54 ..
drwxr-xr-x  2 nobody admin 4096 Feb  9 03:55 dir1
-rw-r--r--  1 nobody admin    0 Feb  9 03:54 file1
-rw-r--r--  1 nobody admin    0 Feb  9 03:55 file2

IMPORTANT: here's one thing which is often forgotten: when you're changing an owner of a directory, this DOES NOT automatically change owner of all the files which already exist in this directory. So, if we check the file3 in dir1 after the example above, we can see that even though dir1 now belongs to user nobody, file3 in it still belongs to me:

ubuntu# ls -l dir1/file3
-rw-r--r-- 1 greys admin 0 Feb  9 03:55 dir1/file3

If your intention is to change ownership of all the files and directories of a certain location in your filesystem, you need to use a -R option of the chown command, which means recursive ownership change:

ubuntu# chown -R nobody dir1
ubuntu# ls -l dir1/file3
-rw-r--r-- 1 nobody admin 0 Feb  9 03:55 dir1/file3

And just to further demonstrate this, I'm going to change owner of all the files and directories in /home/greys/example directory back to my own username, greys:

ubuntu# chown -R greys /home/greys/example/
ubuntu# ls -l /home/greys/example/
total 4
drwxr-xr-x 2 greys admin 4096 Feb  9 03:55 dir1
-rw-r--r-- 1 greys admin    0 Feb  9 03:54 file1
-rw-r--r-- 1 greys admin    0 Feb  9 03:55 file2

That's it for today, good luck with changing file owners on your Unix system!

Recommended books:

See also:

• Finding the owner of a file in Unix
• Find files which belong to a given Unix user
• What to do when numeric user id is shown instead of username
  

How to check if a file exists in Perl

First of all, we need to create one file for our experiments:

ubuntu$ touch /tmp/file.try

Now, create a new file with our script: /tmp/file-try.pl (don't forget to chmod a+x /tmp/file-try.pl afterwards to make it executable):

#!/usr/bin/perl
#
$FILE1="/tmp/file.try";
$FILE2="/tmp/file2.try";
#
if (-e $FILE1) {
        print "File $FILE1 exists!\n";} else {
        print "File $FILE1 is not found!\n";
}
if (-e $FILE2) {
        print "File $FILE2 exists!\n";
} else {
        print "File $FILE2 is not found!\n";
}

Now, as you can see from the example, the actual condition checking for the existence of a file is this:

if (-e $FILE)

where $FILE is the string containing the filename.

In my example, there's two filenames ($FILE1 and $FILE2) and two conditions, so that you can see for yourself how one of them confirms the existence of a file and another one proves that there's no such file found.

To see the script work, simply run it without any parameters:

ubuntu$ /tmp/file-try.pl
File /tmp/file.try exists!
File /tmp/file2.try is not found!

That's all for today! Enjoy your Perl scripting!

See also:

• How to find out file permissions in Perl
• Walking a directory tree in Perl
• Unix glossary
  

Files and directories ownership in Unix

If you look at any file using ls command, you will see an output like the one shown below – it reveals file access permissions, user and group id of the owner, the modification timestamp and the file name itself:

ubuntu$ ls -l /tmp/myfile
-rw-r--r-- 1 greys admin 0 Jan  6 03:51 /tmp/myfile

In this example, the /tmp/myfile file belongs to me, hence the username is greys. It also belongs to my default (primary) Unix group – admin.

Similarly, ownership of any file or any directory can be confirmed for every object in available filesystems. Here's just a few more examples, these are the standard system files belonging to root:

ubuntu$ ls -ald /etc /etc/passwd
drwxr-xr-x 91 root root 4096 Jan  6 03:51 /etc
-rw-r--r--  1 root root 1481 Jan  6 03:51 /etc/passwd

Why numeric IDs are sometimes shown instead of username or groupname

Sometimes though, you will look at a file and instead of the username you will see a numeric ID:

ubuntu# ls -al /tmp/file
-rw-r--r-- 1 1006 root 0 Jan  6 03:51 /tmp/file

The reason numberic ID (1006 in the example above) is shown instead of a username is because your system doesn't recognize this ID – it can't be associated to any username known to your Unix OS.

There are a few possible scenarios for this to happen, but most likely the user has been removed since the file was created. Naturally, deleting any User doesn't automatically mean removing every single file belonging to such a user, that's why the files stay but can no longer be associated with the existing user. All they have to show is the user ID which once owned the file.

How to find the missing username using user id

Unfortunately, there are no easy ways to recover the username (or any other user-specific information) based on a misterious user ID some of your files might have. There are a few things can try though.

1. Try other Unix systems in your environment

   It can be the case that Unix account was a local one automatically created by your system administrators. There's still a chance the same uid exists on other systems. Log into a few of them and verify if they have a user with the same user id (read this post for more information: How to Find Out user id):

   ubuntu$ getent passwd 1006
   newowner:x:1006:1006::/home/newowner:/bin/sh

   The same tip applies in case of more mature environments where Unix systems don't have local users, but instead rely on NIS or LDAP for accessing user accounts information.

   If your system for whatever reason can't access the centralized storage for users, you will experience the same symptoms – most of files belonging to users will appear to have numeric IDs instead of usernames. Most likely though, you'll have more important problems in this case – like not being able to log in as anything else but root (which is an administrative account always created locally on each system).

2. Look at home directories and their owners

   When a new user is added to Unix system, it usually gets a home directory assigned to it. Creating a home directory is not a default behaviour at times, but it's a good practice and so there's a very high probability that the user you're looking for had a home directory. Removing home directories isn't usually done at the same time when a user is removed, so there's also a good chance that even though the user isn't found anymore, the home directory is still there.

   What you should be looking for is a home directory which belongs to the same user id which some of the unidentified files of yours belong to.

   Simply do an ls command under /home directory and see if any of the directories there appear to have numeric IDs instead of usernames:

   ubuntu# ls -ald /home/*
   drwxr-xr-x  2 ftp   nogroup  4096 May 22  2007 /home/ftp
   drwxr-xr-x 11 greys greys    4096 Dec 13 19:56 /home/greys
   drwxr-xr-x  2 1006  admin    4096 Jan  6 04:23 /home/mike

   As you can see, sometimes you might get lucky – the directory is there, and since most home directories usually have the same name as the username which owns them, you can deduct that the username of the user id 1006 was "mike". You can now recreate Mike's account and it will be immediately reflected for all the files owned by user id 1006:

   ubuntu# ls -ald /home/mike
   drwxr-xr-x 2 1006 admin 4096 Jan  6 04:23 /home/mike
   root@simplyunix:~# useradd -u 1006 mike
   root@simplyunix:~# ls -ald /home/mike /tmp/file
   drwxr-xr-x 2 mike admin 4096 Jan  6 04:23 /home/mike
   -rw-r--r-- 1 mike root 0 Jan  6 03:51 /tmp/file

3. Look at other users known to your systemSometimes users are created in batches, and you can guess who the user was by looking which users were created before and after. All you have to do is to use the same getent passwd approach for user ids which are smaller or larger than the one you want to identify.

   Another way to user other users' information to your advantage is to verify which groups they belong to and to then query the groups to see if they have any members not currently known to your system. This will only work for NIS/LDAP groups, not local ones. What could happen is that even though a user was removed, the username is still listed in a few NIS groups.

That's it for today. Hope this post helps you in your investigations, and stay tuned for more!

See also:

• Adding users in Unix
• How to list groups a Unix user belongs to
  
• How to find out user ID

Just to remind you all, rsync is a remote synchronization tool. This means that its primary use lies in the filed of synchronizing files and directories between remote Unix systems, but I feel that it's really important for you to understand the basics before moving on to more advanced stuff.

The beauty of rsync is that its syntax and command line options are exactly the same for local and remote directories synchronizations, so it's not like you'll have to learn it all from scratch when I show you the real world usage of rsync in my next post.

rsync experiments setup

Just use the following commands to re-create your rsync playground for this post:

ubuntu$ rm -rf /tmp/dir1 /tmp/dir2
ubuntu$ mkdir /tmp/dir1 /tmp/dir2
ubuntu$ cd /tmp
ubuntu$ echo "original file 1" > dir1/file1
ubuntu$ echo "original file 2" > dir1/file2
ubuntu$ echo "original file 3" > dir1/file3
ubuntu$ cp dir1/file1 dir2

It will probably be more convenient for you to just download the same commands in form of a script though, so here's the link: rsync-setup.sh

Tracking rsync progress

The larger the directories you're synchronizing with rsync, the longer it will take for the command to finish. Depending on the scale of your task, it can be minutes, long hours or even days before you get the synchronization complete. With this in mind, the importance of progress tracking with rsync should be obvious to you.

Here's how you make rsync report its progress: just use the –progress command line option (in addition to the command line I introduced you to last time):

ubuntu$ rsync -avz --stats --progress /tmp/dir1/ /tmp/dir2
building file list ...
4 files to consider
file2
          16 100%    0.00kB/s    0:00:00 (xfer#1, to-check=1/4)
file3
          16 100%   15.62kB/s    0:00:00 (xfer#2, to-check=0/4)

Number of files: 4
Number of files transferred: 2
Total file size: 48 bytes
Total transferred file size: 32 bytes
Literal data: 32 bytes
Matched data: 0 bytes
File list size: 59
File list generation time: 0.001 seconds
File list transfer time: 0.000 seconds
Total bytes sent: 193
Total bytes received: 64

sent 193 bytes  received 64 bytes  514.00 bytes/sec
total size is 48  speedup is 0.19

Why you should track the progress of rsyncs

The key differences are that you first get a regularly updated message about building a file list. When you run into hundreds of thousands of files, this becomes quite useful:

building file list ...
4 files to consider

Another way to track progress is to see the transfer progress for each of the involved files. For thousands tiny files it won't be all that impressive a feature, but if you're transferring huge files between remote systems, a dynamic transfer progress for each file will be exactly what you need:

file2
          16 100%    0.00kB/s    0:00:00 (xfer#1, to-check=1/4)
file3
          16 100%   15.62kB/s    0:00:00 (xfer#2, to-check=0/4)

That's it, hope you like this new option. I promise to tell you more some other time!

See also:

• Synchronize Directories in Unix with rsync command
• Comparing directories in Unix
• Finding large files and directories