Basic calculations with bc

bc is a very simple command. It takes standard input as an expression and then evaluates this, performing all the necessary calculations and showing you the result. Thus, to quickly sum numbers up or get a result of some other calculation, simply echo the expression and then pipe it out to the bc command:

ubuntu$ echo "1+2" | bc
3

Now, in scripts your calculations with bc are done quite similarly to what we did in bash. Here's an example:

ubuntu$ NUMBER1=1
ubuntu$ NUMBER2=2
ubuntu$ SUM=$(echo "$NUMBER1+$NUMBER2"| bc)
ubuntu$ echo $SUM
3

I told you these calculations would be basic, right? Now onto the more interesting stuff!

Fixed point calculations with bc

Most people learn about bash math limitations when they attempt to do a simple calculation but can't get the current answer with fixed point values. By default, all the operations happen with integers, and that's what you would get:

ubuntu$ echo "1/2" | bc
0

Now, if you expect 0.5 to be the result of dividing 1 by 2, you need to explain it to bc, because by default it doesn't show you any fractional part of the number.

The way you do this is quite simple: all you have to do is specify the number of digits you'd like to see  after the radix point of your result. For example, if I set this number to 5, I'll get bc to output the result of my calculation with 5 digits after the radix point. The special keyword to convey this intention to the bc command is called scale. Just specify the scale value and separate it from the rest of your expression by the semicolon sign:

ubuntu$ echo "scale=5; 1/2" | bc
.50000

Here's another example:

ubuntu$ echo "scale=5; 0.16*10.79" | bc
1.7264

Hope this answers your question! bc command is very powerful, so I'll definitely have to revisit it again in the future. For now though, enjoy the fixed point calculations and be sure to ask questions if you think I can help!

See also:

• Summing numbers up in Unix scripts
• Basic math in Unix scripting
• Another way of doing calculations in scripts
  

Basic construction for summing up in shell scripts

In my Basic arithmetic operations in Unix shell post last year, I've shown you how to sum up two numbers:

SUM=$(($NUMBER1 + $NUMBER2)

using the same approach, it's possible to calculate sums of as many numbers as you like, if you use one of the loops available in your shell.

Before we get started, let's create a simple file with numbers we'll work with:

ubuntu$ for i in 1 2 3 4 5 6 7 8 9; do echo $i >> /tmp/nums; done;
ubuntu$ cat /tmp/nums
1
2
3
4
5
6
7
8
9

Using a while loop to sum numbers up in Unix

Here's an example of how you can use a while loop in Unix shell for summing numbers up. Save it as a /tmp/sum.sh script, and don't forget to do chmod a+x /tmp/sum.sh so that you can run it!

#!/bin/sh
#
SUM=0
#
while read NUM
do
        echo "SUM: $SUM";
        SUM=$(($SUM + $NUM));
        echo "+ $NUM: $SUM";
done < /tmp/nums

Here's how the output will look when you run it:

ubuntu$ /tmp/sum.sh
SUM: 0
+ 1: 1
SUM: 1
+ 2: 3
SUM: 3
+ 3: 6
SUM: 6
+ 4: 10
SUM: 10
+ 5: 15
SUM: 15
+ 6: 21
SUM: 21
+ 7: 28
SUM: 28
+ 8: 36
SUM: 36
+ 9: 45

Of course, your data will most probably will be in a less readable form, so you'll have to do a bit of parsing before you get to sum the numbers up, but the loop will be organized in the same way.

That's it for today, enjoy and feel free to ask questions!

See also:

• Basic arithmetics in scripts
• Variables in Unix shells
• Unix glossary
  

GNU date command advantage

GNU version of the date command, although supporting a common syntax, has one great option: it allows you to specify the desired date with a simple string before reporting it back. What this means is that by default this specified date is assumed to be "now", but you can use other keywords to shift the result of the date command and thus show what date it was yesterday or a week ago:

Here's a default date output for the current date and time:

ubuntu$ date
Fri Sep 19 08:06:41 CDT 2008

Now, the parameter for specifying desired date is -d or –date, and if you use it with the "now" or "today" parameter, you'll get similar output:

ubuntu$ date -d now
Fri Sep 19 08:06:44 CDT 2008
ubuntu$ date -d today
Fri Sep 19 08:06:50 CDT 2008

Showing tomorrow's date

Similarly, you can get tomorrow's date:

ubuntu$ date -d tomorrow
Sat Sep 20 08:02:12 CDT 2008

Obviously, if you feel like specifying a format for the date, you can do it:

ubuntu$ date -d tomorrow "+%b %d, %Y"
Sep 20, 2008

Find out yesterday's date

Again, there's no rocket science involved in showing yesterday's date neither:

ubuntu$ date -d yesterday "+%b %d, %Y"
Sep 18, 2008

Show a date few days away

If you're interested in a certain date a few days or even weeks away, here's how you can do it:

Example 1: a date 5 days ago

ubuntu$ date -d "-5 days"
Sun Sep 14 08:45:57 CDT 2008

Example 2: a day 2 weeks from now

ubuntu$ date -d "2 weeks"
Fri Oct  3 08:45:08 CDT 2008

Example 3: a day two weeks ago from now

ubuntu$ date -d "-2 weeks"
Fri Sep  5 08:45:11 CDT 2008

Extreme example: a day 50 years ago

If you're really curious about dates in Unix, you can even make GNU date go back a few years:

ubuntu$ date -d "-50 years"
Fri Sep 19 08:47:51 CDT 1958

That's it for today, hope you like this little discovery – having mostly worked with Solaris systems most of my career, I didn't know my Ubuntu had this functionality bonus. Really useful!

See also:

• Time and date in Unix scripts
• Variables in Unix shell scripts
  
• Updating shell variables in Unix
  

Use Parenthesis to simplify math expressions

In Basic Arithmetic Operations post I've shown you how expression evaluation can be used to calculate simple math expressions in your Unix shell:

ubuntu$ START=1
ubuntu$ FINISH=10
ubuntu$ ELAPSED=$(($FINISH - $START))
ubuntu$ echo $ELAPSED
9

Although this approach looks clean enough, there's a way to simplify it even further if you put everything in parenthesis. In other words, the same result (ELAPSED receiving a correct value of FINISH value minus START value) can be achieved this way:

ubuntu$ ((ELAPSED=FINISH-START))
ubuntu$ echo $ELAPSED
9

It's a matter of preference, and I used to always do such calculations the former way shown, but now that one of the readers of this blog pointed out the latter way of using it, I think I might change my habits – I actually like this way much better.

See also:

• Time and date in Unix scripts
• Using variables in scripts
  
• Basic math in Unix scripts

Updating shell variable in Unix

If you have a text variable in your script and would like to append some text to it or somehow process this value, it's perfectly normal to reuse the current variable value using a syntax like this:

ubuntu$ echo $VAR1
hello
ubuntu$ VAR1=$VAR1" world!"
ubuntu$ echo $VAR1
hello world!

You see? It's this easy!

Common cases of reusing variable values in Unix shells

Most frequently, I use the technique described above to update some of my environment variables. I'm sure you'll find them useful too.

Updating the PATH variable

PATH is the user environment variable responsible for the directories where Unix shell looks for executable commands every time you type something. Quite often you get a "file not found" error not because there isn't such a command installed in your OS, but simply because your PATH variable has not a corrept path to a directory with that command.

Here's an example from a standard user on one of my Red Hat Enterprise Linux systems.

I like the runlevel command, it's quite simple and can be useful in scripts. When I attempt to run it, here's what happens:

redhat$ runlevel
bash: runlevel: command not found

Like I said earlier, it's most likely because my PATH variable doesn't have the /sbin directory which is where this command resides. Let's confirm this:

redhat$ echo $PATH
/usr/lib/qt-3.3/bin:/usr/kerberos/bin:/usr/local/bin:/bin:/usr/bin:/usr/X11R6/bin

Now it's time to update the PATH and append the /sbin path to it:

redhat$ PATH=$PATH:/sbin

runlevel command will be found now and here's a proof:

redhat$ runlevel
N 5

Updating the MANPATH variable

Another good example is the MANPATH variable, which contains the list of directories with manpages which man command uses.

If some command doesn't have its man page in neither of the directories specified by MANPATH, you'll get an error.

Here's an example from one of the Solaris systems, I'm looking for a Veritas vxprint command man page:

solaris$ man vxprint
No manual entry for vxprint.
solaris$ echo $MANPATH
/usr/man:/opt/sfw/man:/usr/perl5/man

Let's add the /opt/VRTS/man to the MANPATH variable:

bash-2.03# MANPATH=$MANPATH:/opt/VRTS/man
solaris$ echo $MANPATH
/usr/man:/opt/sfw/man:/usr/perl5/man:/opt/VRTS/man

It's bound to work now:

solaris$ man vxprint
Reformatting page.  Please Wait... done
...

That's all I wanted to share with you today. Hope you liked the tip, and as always – feel free to ask any questions!

See also:

• Environment variables in Unix
• Using variables in shell scripts
• Unix scripts: basic arithmetic operations
  

Today I'd like to talk about parsing text files in Unix shell scripts. This is one of the really popular areas of scripting, and there's a few quite typical limitations which everyone comes across.

Reading text files in Unix shell

If we agree that by "reading a text file" we assume a procedure of going through all the lines found in a clear text file with a view to somehow process the data, then cat command would be the simplest demonstration of such procedure:

redhat$ cat /etc/redhat-release
Red Hat Enterprise Linux Client release 5 (Tikanga)

As you can see, there's only one line in the /etc/redhat-release file, and we see what this line is.

But if you for whatever reason wanted to read this file from a script and assign the whole release information line to a Unix variable, using cat output would not work as expected:

bash-3.1$ for i in `cat /etc/redhat-release`; do echo $i; done;
RedHat
Enterprise
Linux
Client
release
5
(Tikanga)

Instead of reading a line of text from the file, our one-liner splits the line and outputs every word on a separate line of the output. This happens because of the shell syntax parsing – Unix shells assume space to be a delimiter of various elements in a list, so when you do a for loop, Unix shell interpreter treats each line with spaces as a list of elements, splits it and returns elements one by one.

How to read text files line by line

Here's what I decided: if I can't make Unix shell ignore the spaces between words of each line of text, I'll disguise these spaces. Since my solution was getting pretty bulky for a one-liner, I've made it into a script. Here it is:

bash-3.1$ cat /tmp/cat.sh
#!/bin/sh
FILE=$1
UNIQUE='-={GR}=-'
#
if [ -z "$FILE" ]; then
        exit;
fi;
#
for LINE in `sed "s/ /$UNIQUE/g" $FILE`; do
        LINE=`echo $LINE | sed "s/$UNIQUE/ /g"`;
        echo $LINE;
done;

As you can see, I've introduced an idea of a UNIQUE variable, something containing a unique combination of characters which I can use to replace spaces in the original string. This variable needs to be a unique combination in a context of your text files, because later we turn the string back into its original version, replacing all the instances of $UNIQUE text with plain spaces.

Since most of the needs of mine required such functionality for a relatively small text files, this rather expensive (in terms of CPU cycles) approach proved to be quite usable and pretty fast.

Update: please see comments to this post for a much better approach to the same problem. Thanks again, Nails!

Here's how my script would work on the already known /etc/redhat-release file:

bash-3.1$ /tmp/cat.sh /etc/redhat-release
Red Hat Enterprise Linux Client release 5 (Tikanga)

Exactly what I wanted! Hopefully this little trick will save some of your time as well. Let me know if you like it or know an even better one yourself!

Related books

If you want to learn more, here's a great book:

Classic Shell Scripting

See also:

• Using variables in scripts
• Unix scripting: time and date

Going further, I'd like to show you some basics of working with time and date in your scripts – generating all sorts of timestamps and timing some parts of your script for reporting purposes.

The history of (Unix) time

In case you didn't know, time in Unix starts with a Unix epoch, sometimes also referred to as POSIX epoch. The Unix epoch is the time 00:00:00 UTC on January 1, 1970.

Most of Unix-like system today store and calculate this time according to Unix epoch, which means they all have an internal counter which counts the number of seconds elapsed since midnight of January 1, 1970.

As you can imagine, this is a pretty big number (it passed 1,000,000,000 seconds back in 2001, just so that you know), and reporting in directly would not be very useful to real humans. Because of this, most of Unix time and date reporting commands and functions do the conversion from Unix time into something more meaningful to the end user.

While for the vast majority of time and date related tasks this conversion is good, a task of timing certain events can sometimes benefit from using raw second counters and simply subtracting them when needed.

Getting Unix time and date

In Unix shells, the easiest way to get a current time and date is to use the date command:

ubuntu$ date
Tue Jun 10 10:46:07 IST 2008

date is a very smart command, and apart from this default behavior it supports template system for printing the current time and date – so you can use it to report only specific part of the time and date like the current hour or the day of a week or the year.

Read the man page for date (type man date in your shell prompt) to learn all the details, but this is how you use date output templates:

ubuntu$ date "+%b %d, %Y"
Jun 10, 2008

In this example, the %b parameter in this template represents the short name of the current month, %d is the day of the month, and %Y is the four-digit representation of the current year.

Timing parts of your Unix script

If you're after timing some parts of your script, you'll need to use two variables: one for saving the time before the start of an observed part of the script, and another one for saving the time after the same piece of code.

These two variables will be used to subtract the time and report the elapsed time in seconds, so in order to do this we'll need the date command to report time in seconds. That's why I've given you an introduction to Unix time earlier: date reports the number of seconds since the Unix epoch:

ubuntu$ date +%s
1213091896

If you run it just a few seconds later, you'll see a different number:

ubuntu$ date +%s
1213091922

That's why, if such values are saved and then subtracted, we'll get the elapsed time in seconds.

Here's a simple script showing how this is done:

#!/bin/bash
#
START=`date +%s`
echo "Script start time (Unix epoch): $START"
#
echo "- sleeping for 3 seconds..."
sleep 3
echo "- sleeping for 2 seconds more..."
sleep 2
#
FINISH=`date +%s`
echo "Script finish time (Unix epoch): $START"
#
ELAPSED=`expr $FINISH - $START`
echo "Elapsed time: $ELAPSED"

And if you run it, you will see this output:

ubuntu$ /tmp/time-example.sh
Script start time (Unix epoch): 1213092131
- sleeping for 3 seconds...
- sleeping for 2 seconds more...
Script finish time (Unix epoch): 1213092131
Elapsed time: 5

That's all I wanted to show you today. In future posts, I'll show you a few more things you can do regarding timing and timestamps in Unix. Until then – good luck with your scripting, and feel free to ask if you need any more help!

Related books

If you want to learn more, here's a great book:

Mastering Unix Shell Scripting

See also:

• Unix time – a great article on Wikipedia
• date command in Unix
  
• Working with variables in Unix shell

Defining a variable in Unix shell

To specify a value for a variable, you need to decide on the variable name – can be any word or combination of English alphabet symbols and digits, and specify the value.

In Bourne shell (sh), Bourne Again Shell (bash) and Korn Shell (ksh), here's how you would define a new variable and assign it a value:

CONFIG_FILE="/etc/myfile"

In C-Shell (csh), it's done like this:

setenv CONFIG_FILE "/etc/myfile"

Basic variables usage

To access the value of a variable, you need to use the same variable name, but with a dollar sign in front of it, like this:

$CONFIG_FILE

Important: to set a new value to the variable, you use just the variable name like CONFIG_FILE, but to access this value later you need to use the $CONFIG_FILE form.

The most basic way to use variables is to assign them constant values at the beginning of your script. This is a good way to define locations of standard files or directories your script will work with, etc:

#!/bin/sh
#
CONFIG_FILE=/etc/myfile
MY_DIR=/etc
echo $CONFIG_FILE

Using output of Unix commands to set variables

One of the best things about shell scripting is that it's very easy to use any Unix command to generate the output and use it to set the variable.

In this example, I'm running a date command and saving its output as values for my variables:

ubuntu$ cat /tmp/1.sh

#!/bin/sh
#
STARTED=`date`
sleep 5
FINISHED=`date`
#
echo "Script start time: $STARTED"
echo "Script finish time: $FINISHED"

If I run this simple script, I see the following:

ubuntu$ /tmp/1.sh
Script start time: Wed May 7 04:56:51 CDT 2008
Script finish time: Wed May 7 04:56:56 CDT 2008
The same approach can be used for practically any scenario.

Here's an example of using uname command to extract some useful information about our system:

#!/bin/sh
#
STARTED=`date`
NODE=`uname -n`
OS=`uname -o`
CPU=`uname -p`
FINISHED=`date`
#
echo "Nodename: $NODE"
echo "OS type: $OS"
echo "Processor: $CPU"
echo "Script start time: $STARTED"
echo "Script finish time: $FINISHED"

And this is how it works:

ubuntu$ /tmp/1.sh
Nodename: ubuntu
OS type: GNU/Linux
Processor: unknown
Script start time: Wed May  7 05:05:31 CDT 2008
Script finish time: Wed May  7 05:05:31 CDT 2008

That's it for today! Let me know what next topic about Unix shell scripting you'd like to see covered, and I'll do my best to explain it in the coming posts.

Related books

If you want to learn more, here's a great book:

Linux Command Line Shell Scripting Bible

find /directory -ls

command which I normally use for such tasks wasn't terribly useful because there was no space delimiter between a username and a group. Results of such scan of the directory tree will have to later be parsed by other tools, and that's why proper splitting of the output into separate fields is so important.

This issue was motivational enough to refresh my Perl skills and sketch the following script (based entirely on this Never Run Unix Find Again article).

It's a very simple piece of code which takes a directory to scan as a parameter.

How this works

As you can see, we're using the standard File:Find functionality, and the two parameters find function takes are the wanted function, where you put conditions for your search.

Within this function, you call lstat to obtain all the necessary information about each directory entry, and then output the necessary fields.

Perl code

#!/usr/bin/perl
use File::Find;
if ($ARGV[0] ne "") {
        $dir = $ARGV[0];
} else {
        print "Please specify a directory!";
        exit;
}

find &wanted, $dir;

sub wanted {
  my $dev;         # the file system device number
  my $ino;         # inode number
  my $mode;        # mode of file
  my $nlink;       # counts number of links to file
  my $uid;         # the ID of the file's owner
  my $gid;         # the group ID of the file's owner
  my $rdev;        # the device identifier
  my $size;        # file size in bytes
  my $atime;       # last access time
  my $mtime;       # last modification time
  my $ctime;       # last change of the mode
  my $blksize;     # block size of file
  my $blocks;      # number of blocks in a file
  my $user;	# username
  my $group;	# unix group name

#Right below here your telling lstat to retrieve all this info on each and every file/directory. Each and every file/directory is written to $_.

  (($dev,$ino,$mode,$nlink,$uid,$gid,$rdev,$size,$atime,$mtime,$ctime,$blksize,$blocks) = lstat($_));
  $user = getpwuid($uid);
  $group = getgrgid($gid);

  print $File::Find::name . ":$mode:$size:$user:$group:$ctime:$mtimen";
}

Hope you find this useful. Good luck with finding all your files!

For further reading, please consult the Perldoc section on File:Find.