How to confirm what Unix groups are available

If you remember, a while ago I've introduced you to the getent command. It's a great way of querying various information databases about your systems' users, groups and some other objects. Here's how you would use the command to get a full list of Unix groups known to your system:

ubuntu# getent group
root:x:0:
daemon:x:1:
bin:x:2:
sys:x:3:
adm:x:4:
tty:x:5:
disk:x:6:
lp:x:7:
mail:x:8:
news:x:9:
uucp:x:10:
man:x:12:
proxy:x:13:
kmem:x:15:
dialout:x:20:
fax:x:21:
...

This is an abridged output, but I hope you get the idea. This output helps you confirm the following:

1. Unix group name – first field
2. Unix group ID (gid) – third field
3. All the usernames of members for various groups – fourh field, unfortunatelly empty for all the groups in my example.

How to confirm the members of a Unix group

Using the same getent command, you can query the groups database using a group name. In my example below, I'm confirming the membership of a mygroup Unix group:

ubuntu# getent group mygroup
mygroup:x:1002:user1,greys,user2

How to determine the number of Unix groups known to your system

One more thing you can learn about your Unix groups using getent command is to confirm the overall number of Unix groups – some scenarious require you to have this number. Here's how you would use getent together with the wc command to confirm the number of groups:

ubuntu# getent group | wc -l
62

See also:

Why am I asking these questions?

As you remember, a month ago I have offered invited all of the Unix Tutorial readers to learn Unix together. Everyone benefits from this – you get a chance to ask the questions which you always wanted answered, and I get to refresh my mind or even conduct a research on new topics just so that I can share the answers and solutions in the easiest to follow form.

I'm currently working on a members area for Unix Tutorial, which will eventually have a number of self-paced courses to help you improve your knowledge of Unix and get to the next level of productivity when solving technical problems.

Update: if you're interested in becoming a member, subscribe to the Unix Tutorial waiting list!

But guess what? Without knowing what exactly you do and why you ask the questions about Unix, it's extremely hard to address some of the topics. I would like to give the fullest coverage to everything I present, but at the same time it probably wouldn't make sense to spend much time explaining some concepts most of you are familiar with already.

With this in mind, I'd like you to please take a few moments and help me out by sharing a bit about yourself and your experience with Unix-like operating systems so far. Please feel free to leave comments or use a contact form – any form of your help will be greatly appreciated.

Where are you using Unix?

More likely than not, you have arrived at one of the Unix Tutorial pages while searching for an answer. Webserver logs give me a pretty good idea about your interests and challenges, but they don't reveal one important piece of information: where do you use your Unix?

• Are you a home user with Ubuntu desktop?
• Do you have a RedHat Linux desktop at your workplace?
• Is the hospital you work for using Unix-based equipment?
• Do you have Linux on your PDA or MP3 player?

These and many more questions are the ones I'd really like to have answered  – the nature of my technical tips so far had been rather basic, but some of the comments (and especially the email questions submitted through Unix Tutorial contact form) are so technical that it's obvious their authors have many years of Unix experience.

How are you using Unix?

This is another question which really interests me. Recent years saw many Unix-like OS distributions rapidly advance to take up new positions in business solutions of all fields. You can find Unix-like OS in storage and network appliances, desktops, servers, laptops and netbooks, MP3 players and book readers. It's hard to imagine a technological niche which cannot benefit from using a Unix-like system.

So how do you use your Unix?

• Do you manage Unix servers in your hosting startup?
• Do you develop software on a Linux platform?
• Are you using Unix in your finance department?
• Is Unix-based solution behind your compute grid used for some kind of research?
• Are you a professional artist or photographer working in Unix?
• Do you compose music or produce podcasts and vidcasts?

Let me know about all the ways you've found Unix to be useful so far – I'm quite often surprised to discover new ways people manage to use the most unusual operating systems, so there's bound to be plenty of interesting ideas about Unix!

Why are you using Unix?

This is the last question for today.  Glad to still have you with me, reading this post! I REALLY appreciate your time and willingness to help!

Why are you a Unix user at the moment? Is it because:

• it's a requirement at work
• it's a great family of operating systems which you simply love working with
• you're a hacker or a developer
• Unix-like OS is the only option you have for the unique task at hand
• you've been told Unix is cool
• you've been told Unix is so not cool, but got curious and simply couldn't resist

Whatever your reason may be, I'd like to know it – please share your thoughts!

Thank you!

Thanks for stopping by and reading through all the questions! I will be delighted to hear back from you about your Unix story, and if there are any questions you decide to ask me – I'll be happy to answer.

See also:

• Join the Unix Tutorial waiting list

Files and directories ownership in Unix

If you look at any file using ls command, you will see an output like the one shown below – it reveals file access permissions, user and group id of the owner, the modification timestamp and the file name itself:

ubuntu$ ls -l /tmp/myfile
-rw-r--r-- 1 greys admin 0 Jan  6 03:51 /tmp/myfile

In this example, the /tmp/myfile file belongs to me, hence the username is greys. It also belongs to my default (primary) Unix group – admin.

Similarly, ownership of any file or any directory can be confirmed for every object in available filesystems. Here's just a few more examples, these are the standard system files belonging to root:

ubuntu$ ls -ald /etc /etc/passwd
drwxr-xr-x 91 root root 4096 Jan  6 03:51 /etc
-rw-r--r--  1 root root 1481 Jan  6 03:51 /etc/passwd

Why numeric IDs are sometimes shown instead of username or groupname

Sometimes though, you will look at a file and instead of the username you will see a numeric ID:

ubuntu# ls -al /tmp/file
-rw-r--r-- 1 1006 root 0 Jan  6 03:51 /tmp/file

The reason numberic ID (1006 in the example above) is shown instead of a username is because your system doesn't recognize this ID – it can't be associated to any username known to your Unix OS.

There are a few possible scenarios for this to happen, but most likely the user has been removed since the file was created. Naturally, deleting any User doesn't automatically mean removing every single file belonging to such a user, that's why the files stay but can no longer be associated with the existing user. All they have to show is the user ID which once owned the file.

How to find the missing username using user id

Unfortunately, there are no easy ways to recover the username (or any other user-specific information) based on a misterious user ID some of your files might have. There are a few things can try though.

1. Try other Unix systems in your environment

   It can be the case that Unix account was a local one automatically created by your system administrators. There's still a chance the same uid exists on other systems. Log into a few of them and verify if they have a user with the same user id (read this post for more information: How to Find Out user id):

   ubuntu$ getent passwd 1006
   newowner:x:1006:1006::/home/newowner:/bin/sh

   The same tip applies in case of more mature environments where Unix systems don't have local users, but instead rely on NIS or LDAP for accessing user accounts information.

   If your system for whatever reason can't access the centralized storage for users, you will experience the same symptoms – most of files belonging to users will appear to have numeric IDs instead of usernames. Most likely though, you'll have more important problems in this case – like not being able to log in as anything else but root (which is an administrative account always created locally on each system).

2. Look at home directories and their owners

   When a new user is added to Unix system, it usually gets a home directory assigned to it. Creating a home directory is not a default behaviour at times, but it's a good practice and so there's a very high probability that the user you're looking for had a home directory. Removing home directories isn't usually done at the same time when a user is removed, so there's also a good chance that even though the user isn't found anymore, the home directory is still there.

   What you should be looking for is a home directory which belongs to the same user id which some of the unidentified files of yours belong to.

   Simply do an ls command under /home directory and see if any of the directories there appear to have numeric IDs instead of usernames:

   ubuntu# ls -ald /home/*
   drwxr-xr-x  2 ftp   nogroup  4096 May 22  2007 /home/ftp
   drwxr-xr-x 11 greys greys    4096 Dec 13 19:56 /home/greys
   drwxr-xr-x  2 1006  admin    4096 Jan  6 04:23 /home/mike

   As you can see, sometimes you might get lucky – the directory is there, and since most home directories usually have the same name as the username which owns them, you can deduct that the username of the user id 1006 was "mike". You can now recreate Mike's account and it will be immediately reflected for all the files owned by user id 1006:

   ubuntu# ls -ald /home/mike
   drwxr-xr-x 2 1006 admin 4096 Jan  6 04:23 /home/mike
   root@simplyunix:~# useradd -u 1006 mike
   root@simplyunix:~# ls -ald /home/mike /tmp/file
   drwxr-xr-x 2 mike admin 4096 Jan  6 04:23 /home/mike
   -rw-r--r-- 1 mike root 0 Jan  6 03:51 /tmp/file

3. Look at other users known to your systemSometimes users are created in batches, and you can guess who the user was by looking which users were created before and after. All you have to do is to use the same getent passwd approach for user ids which are smaller or larger than the one you want to identify.

   Another way to user other users' information to your advantage is to verify which groups they belong to and to then query the groups to see if they have any members not currently known to your system. This will only work for NIS/LDAP groups, not local ones. What could happen is that even though a user was removed, the username is still listed in a few NIS groups.

That's it for today. Hope this post helps you in your investigations, and stay tuned for more!

See also:

• Adding users in Unix
• How to list groups a Unix user belongs to
  
• How to find out user ID

Today's Unix question is this:

How can we write a shell script in unix to find the day of the week when date is given?

The solution for this is even simpler: there's no need for Unix scripting, all you need is to have GNU date command at your disposal. I've already shown you all the basic date/time calculations using this great tool, and that's just another way of using it.

How to find a Day of the week based on timestamp

All you need is to know the base date. Let's say I'm interested in October 16th, 2009. Here's how easy it is to confirm that day will be Friday:

ubuntu$ date -d "Oct 16 2009" "+%a"
Fri

That's it – enjoy!

See also:

• time and date in Unix
• date calculations with GNU date command
  

Today I'd like to talk about parsing text files in Unix shell scripts. This is one of the really popular areas of scripting, and there's a few quite typical limitations which everyone comes across.

Reading text files in Unix shell

If we agree that by "reading a text file" we assume a procedure of going through all the lines found in a clear text file with a view to somehow process the data, then cat command would be the simplest demonstration of such procedure:

redhat$ cat /etc/redhat-release
Red Hat Enterprise Linux Client release 5 (Tikanga)

As you can see, there's only one line in the /etc/redhat-release file, and we see what this line is.

But if you for whatever reason wanted to read this file from a script and assign the whole release information line to a Unix variable, using cat output would not work as expected:

bash-3.1$ for i in `cat /etc/redhat-release`; do echo $i; done;
RedHat
Enterprise
Linux
Client
release
5
(Tikanga)

Instead of reading a line of text from the file, our one-liner splits the line and outputs every word on a separate line of the output. This happens because of the shell syntax parsing – Unix shells assume space to be a delimiter of various elements in a list, so when you do a for loop, Unix shell interpreter treats each line with spaces as a list of elements, splits it and returns elements one by one.

How to read text files line by line

Here's what I decided: if I can't make Unix shell ignore the spaces between words of each line of text, I'll disguise these spaces. Since my solution was getting pretty bulky for a one-liner, I've made it into a script. Here it is:

bash-3.1$ cat /tmp/cat.sh
#!/bin/sh
FILE=$1
UNIQUE='-={GR}=-'
#
if [ -z "$FILE" ]; then
        exit;
fi;
#
for LINE in `sed "s/ /$UNIQUE/g" $FILE`; do
        LINE=`echo $LINE | sed "s/$UNIQUE/ /g"`;
        echo $LINE;
done;

As you can see, I've introduced an idea of a UNIQUE variable, something containing a unique combination of characters which I can use to replace spaces in the original string. This variable needs to be a unique combination in a context of your text files, because later we turn the string back into its original version, replacing all the instances of $UNIQUE text with plain spaces.

Since most of the needs of mine required such functionality for a relatively small text files, this rather expensive (in terms of CPU cycles) approach proved to be quite usable and pretty fast.

Update: please see comments to this post for a much better approach to the same problem. Thanks again, Nails!

Here's how my script would work on the already known /etc/redhat-release file:

bash-3.1$ /tmp/cat.sh /etc/redhat-release
Red Hat Enterprise Linux Client release 5 (Tikanga)

Exactly what I wanted! Hopefully this little trick will save some of your time as well. Let me know if you like it or know an even better one yourself!

Related books

If you want to learn more, here's a great book:

Classic Shell Scripting

See also:

• Using variables in scripts
• Unix scripting: time and date