Guide to Advanced Linux Command Mastery

Published August 2006

In Sheryl Calish's excellent article “Guide to Linux File Command Mastery," you learned some routine Linux commands, which are especially valuable for Linux newbies. But now that you have mastered the basics, let’s move on to some more sophisticated commands that you will find extremely useful.

In this four-part series, you will learn some not-so-well-known tricks about various routine commands as well as variations in usage that make them more useful. As the series progresses, you will learn successively difficult commands to master.

Note that these commands may differ based on the specific version of Linux you use or which specific kernel is compiled, but if so, probably only slightly.

Painless Changes to Owner, Group, and Permissions

In Sheryl's article you learned how to use chown and chgrp commands to change ownership and group of the files. Say you have several files like this:

# ls -l
total 8
-rw-r--r--    1 ananda   users          70 Aug  4 04:02 file1
-rwxr-xr-x    1 oracle   dba           132 Aug  4 04:02 file2
-rwxr-xr-x    1 oracle   dba           132 Aug  4 04:02 file3
-rwxr-xr-x    1 oracle   dba           132 Aug  4 04:02 file4
-rwxr-xr-x    1 oracle   dba           132 Aug  4 04:02 file5
-rwxr-xr-x    1 oracle   dba           132 Aug  4 04:02 file6

and you need to change the permissions of all the files to match those of file1. Sure, you could issue chmod 644 * to make that change—but what if you are writing a script to do that, and you don’t know the permissions beforehand? Or, perhaps you are making several permission changes and based on many different files and you find it infeasible to go though the permissions of each of those and modify accordingly.

A better approach is to make the permissions similar to those of another file. This command makes the permissions of file2 the same as file1:

chmod --reference file1 file2 

Now if you check:

# ls -l file[12]
total 8
-rw-r--r--    1 ananda   users          70 Aug  4 04:02 file1
-rw-r--r--    1 oracle   dba           132 Aug  4 04:02 file2

The file2 permissions were changed exactly as in file1. You didn’t need to get the permissions of file1 first.

You can also use the same trick in group membership in files. To make the group of file2 the same as file1, you would issue:

# chgrp --reference file1 file2
# ls -l file[12]
-rw-r--r--    1 ananda   users          70 Aug  4 04:02 file1
-rw-r--r--    1 oracle   users         132 Aug  4 04:02 file2

Of course, what works for changing groups will work for owner as well. Here is how you can use the same trick for an ownership change. If permissions are like this:

# ls -l file[12] 
-rw-r--r--    1 ananda   users          70 Aug  4 04:02 file1
-rw-r--r--    1 oracle   dba           132 Aug  4 04:02 file2

You can change the ownership like this:

# chown --reference file1 file2
# ls -l file[12] 
-rw-r--r--    1 ananda   users          70 Aug  4 04:02 file1
-rw-r--r--    1 ananda   users         132 Aug  4 04:02 file2

Note that the group as well as the owner have changed.

More on Files

The ls command, with its many arguments, provides some very useful information on files. A different and less well known command – stat – offers even more useful information.

Here is how you can use it on the executable “oracle”, found under $ORACLE_HOME/bin.

# cd $ORACLE_HOME/bin
# stat oracle
  File: `oracle'
  Size: 93300148        Blocks: 182424     IO Block: 4096   Regular File
Device: 343h/835d       Inode: 12009652    Links: 1    
Access: (6751/-rwsr-s--x)  Uid: (  500/  oracle)   Gid: (  500/     dba)
Access: 2006-08-04 04:30:52.000000000 -0400
Modify: 2005-11-02 11:49:47.000000000 -0500
Change: 2005-11-02 11:55:24.000000000 -0500

Note the information you got from this command: In addition to the usual filesize (which you can get from ls -l anyway), you got the number of blocks this file occupies. The typical Linux block size is 512 bytes, so a file of 93,300,148 bytes would occupy (93300148/512=) 182226.85 blocks. Since blocks are used in full, this file uses some whole number of blocks. Instead of making a guess, you can just get the exact blocks.

You also get from the output above the GID and UID of the ownership of the file and the octal representation of the permissions (6751). If you want to reinstate it back to the same permissions it has now, you could use chmod 6751 oracle instead of explicitly spelling out the permissions.

The most useful part of the above output is the file access timestamp information. It shows you that the file was accessed on 2006-08-04 04:30:52 (as shown next to “Access:”), or August 4, 2006 at 4:30:52 AM. This is when someone started to use the database. The file was modified on 2005-11-02 11:49:47 (as shown next to Modify:). Finally, the timestamp next to “Change:” shows when the status of the file was changed.

-f, a modifier to the stat command, shows the information on the filesystem instead of the file:

# stat -f oracle
  File: "oracle"
    ID: 0        Namelen: 255     Type: ext2/ext3
Blocks: Total: 24033242   Free: 15419301   Available: 14198462   Size: 4096
Inodes: Total: 12222464   Free: 12093976  

Another option, -t, gives exactly the same information but on one line:

# stat -t oracle 
oracle 93300148 182424 8de9 500 500 343 12009652 1 0 0 1154682061 
1130950187 1130950524 4096

This is very useful in shell scripts where a simple cut command can be used to extract the values for further processing.

# stat sqlplus*
  File: 'sqlplus'
  Size: 9865            Blocks: 26         IO Block: 4096   Regular File
Device: 343h/835d       Inode: 9126079     Links: 1    
Access: (0751/-rwxr-x--x)  Uid: (  500/  oracle)   Gid: (  500/     dba)
Access: 2006-08-04 05:15:18.000000000 -0400
Modify: 2006-08-04 05:15:18.000000000 -0400
Change: 2006-08-04 05:15:18.000000000 -0400
 
  File: 'sqlplusO'
  Size: 8851            Blocks: 24         IO Block: 4096   Regular File
Device: 343h/835d       Inode: 9125991     Links: 1    
Access: (0751/-rwxr-x--x)  Uid: (  500/  oracle)   Gid: (  500/     dba)
Access: 2006-08-04 05:13:57.000000000 -0400
Modify: 2005-11-02 11:50:46.000000000 -0500
Change: 2005-11-02 11:55:24.000000000 -0500

File Types

When you see a file, how do you know what type of file it is? The command file tells you that. For instance:

# file alert_DBA102.log
alert_DBA102.log: ASCII text

The file alert_DBA102.log is an ASCII text file. Let’s see some more examples:

# file initTESTAUX.ora.Z
initTESTAUX.ora.Z: compress'd data 16 bits

This tells you that the file is a compressed file, but how do you know the type of the file was compressed? One option is to uncompress it and run file against it; but that would make it virtually impossible. A cleaner option is to use the parameter -z:

# file -z initTESTAUX.ora.Z
initTESTAUX.ora.Z: ASCII text (compress'd data 16 bits)

Another quirk is the presence of symbolic links:

# file spfile+ASM.ora.ORIGINAL   
spfile+ASM.ora.ORIGINAL: symbolic link to 
/u02/app/oracle/admin/DBA102/pfile/spfile+ASM.ora.ORIGINAL

This is useful; but what type of file is that is being pointed to? Instead of running file again, you can use the option -l:

# file -L spfile+ASM.ora.ORIGINAL
spfile+ASM.ora.ORIGINAL: data

This clearly shows that the file is a data file. Note that the spfile is a binary one, as opposed to init.ora; so the file shows up as data file.

file -Lz * | grep ASCII | cut -d":" -f1 | xargs ls -ltr

Comparing Files

How do you find out if two files—file1 and file2—are identical? There are several ways and each approach has its own appeal.

diff. The simplest command is diff, which shows the difference between two files. Here are the contents of two files:

# cat file1
In file1 only
In file1 and file2
# cat file2
In file1 and file2
In file2 only

If you use the diff command, you will be able to see the difference between the files as shown below:

# diff file1 file2
1d0
< In file1 only
2a2
> In file2 only
#

In the output, a "<" in the first column indicates that the line exists on the file mentioned first,—that is, file1. A ">" in that place indicates that the line exists on the second file (file2). The characters 1d0 in the first line of the output shows what must be done in sed to operate on the file file1 to make it same as file2.

Another option, -y, shows the same output, but side by side:

# diff -y file1 file2 -W 120
In file1 only                             <
In file1 and file2                             In file1 and file2
                                          >    In file2 only

The -W option is optional; it merely instructs the command to use a 120-character wide screen, useful for files with long lines.

If you just want to just know if the files differ, not necessarily how, you can use the -q option.

# diff -q file3 file4
# diff -q file3 file2
Files file3 and file2 differ

Files file3 and file4 are the same so there is no output; in the other case, the fact that the files differ is reported.

If you are writing a shell script, it might be useful to produce the output in such a manner that it can be parsed. The -u option does that:

# diff -u file1 file2        
--- file1       2006-08-04 08:29:37.000000000 -0400
+++ file2       2006-08-04 08:29:42.000000000 -0400
@@ -1,2 +1,2 @@
-In file1 only
 In file1 and file2
+In file2 only

The output shows contents of both files but suppresses duplicates, the + and - signs in the first column indicates the lines in the files. No character in the first column indicates presence in both files.

The command considers whitespace into consideration. If you want to ignore whitespace, use the -b option. Use the -B option to ignore blank lines. Finally, use -i to ignore case.

The diff command can also be applied to directories. The command

diff dir1 dir2

shows the files present in either directories; whether files are present on one of the directories or both. If it finds a subdirectory in the same name, it does not go down to see if any individual files differ. Here is an example:

# diff DBA102 PROPRD     
Common subdirectories: DBA102/adump and PROPRD/adump
Only in DBA102: afiedt.buf
Only in PROPRD: archive
Only in PROPRD: BACKUP
Only in PROPRD: BACKUP1
Only in PROPRD: BACKUP2
Only in PROPRD: BACKUP3
Only in PROPRD: BACKUP4
Only in PROPRD: BACKUP5
Only in PROPRD: BACKUP6
Only in PROPRD: BACKUP7
Only in PROPRD: BACKUP8
Only in PROPRD: BACKUP9
Common subdirectories: DBA102/bdump and PROPRD/bdump
Common subdirectories: DBA102/cdump and PROPRD/cdump
Only in PROPRD: CreateDBCatalog.log
Only in PROPRD: CreateDBCatalog.sql
Only in PROPRD: CreateDBFiles.log
Only in PROPRD: CreateDBFiles.sql
Only in PROPRD: CreateDB.log
Only in PROPRD: CreateDB.sql
Only in DBA102: dpdump
Only in PROPRD: emRepository.sql
Only in PROPRD: init.ora
Only in PROPRD: JServer.sql
Only in PROPRD: log
Only in DBA102: oradata
Only in DBA102: pfile
Only in PROPRD: postDBCreation.sql
Only in PROPRD: RMANTEST.sh
Only in PROPRD: RMANTEST.sql
Common subdirectories: DBA102/scripts and PROPRD/scripts
Only in PROPRD: sqlPlusHelp.log
Common subdirectories: DBA102/udump and PROPRD/udump

Note that the common subdirectories are simply reported as such but no comparison is made. If you want to drill down even further and compare files under those subdirectories, you should use the following command:

diff -r dir1 dir2

This command recursively goes into each subdirectory to compare the files and reports the difference between the files of the same names.

# sqlplus / as sysdba
SQL> shutdown immediate
SQL> exit
# export ORACLE_HOME=/u01/app/oracle/product/10.2/db2
# export PATH=$ORACLE_HOME/bin:$PATH
# sqlplus / as sysdba
SQL> @$ORACLE_HOME/rdbms/admin/catalog
...

diff -r /u01/app/oracle/product/10.2/db1 /u01/app/oracle/product/10.2/db2 | 
grep -v Common

cmp. The command cmp is similar to diff:

# cmp file1 file2   
file1 file2 differ: byte 10, line 1

The output comes back as the first sign of difference. You can use this to identify where the files might be different. Like diff, cmp has a lot of options, the most important being the -s option, that merely returns a code:

• 0, if the files are identical
• 1, if they differ
• Some other non-zero number, if the comparison couldn’t be made

Here is an example:

# cmp -s file3 file4
# echo $?
0

The special variable $? indicates the return code from the last executed command. In this case it’s 0, meaning the files file1 and file2 are identical.

# cmp -s file1 file2
# echo $?
1

means file1 and file2 are not the same.

This property of cmp can prove very useful in shell scripting where you merely want to check if two files differ in any way, but not necessarily check what the difference is. Another important use of this command is to compare binary files, where diff may not be reliable.

# cmp sqlplus sqlplusO
sqlplus sqlplusO differ: byte 657, line 7

# ls -l sqlplus*
-rwxr-x--x    1 oracle   dba          8851 Aug  4 05:15 sqlplus
-rwxr-x--x    1 oracle   dba          8851 Nov  2  2005 sqlplusO

comm. The command comm is similar to the others but the output comes in three columns, separated by tabs. Here is an example:

# comm file1 file2
        In file1 and file2
In file1 only
In file1 and file2
        In file2 only

This command is useful when you may want to see the contents of a file not in the other, not just a difference—sort of a MINUS utility in SQL language. The option -1 suppresses the contents found in first file:

# comm -1 file1 file2
In file1 and file2
In file2 only

md5sum. This command generates a 32-bit MD5 hash value of the files:

# md5sum file1
ef929460b3731851259137194fe5ac47  file1

Two files with the same checksum can be considered identical. However, the usefulness of this command goes beyond just comparing files. It can also provide a mechanism to guarantee the integrity of the files.

Suppose you have two important files—file1 and file2—that you need to protect. You can use the --check option check to confirm the files haven't changed. First, create a checksum file for both these important files and keep it safe:

# md5sum file1 file2 > f1f2 

Later, when you want to verify that the files are still untouched:

# md5sum --check f1f2      
file1: OK
file2: OK

This shows clearly that the files have not been modified. Now change one file and check the MD5:

# cp file2 file1
# md5sum --check f1f2
file1: FAILED
file2: OK
md5sum: WARNING: 1 of 2 computed checksums did NOT match

The output clearly shows that file1 has been modified.

Conclusion

Thus far you have learned only some of the Linux commands you will find useful for performing your job effectively. In the next installment, I will describe some more sophisticated but useful commands, such as strace, whereis, renice, skill, and more.