BASH SCRIPT BASICS
1.1. Shell programs
General shell functions –
The UNIX shell program interprets user commands, which are either directly entered by the user, or which can be read from a file called the shell script or shell program. Shell scripts are interpreted, not compiled. The shell reads commands from the script line per line and searches for those commands on the system, while a compiler converts a program into machine readable form, an executable file – which may then be u
in a shell script.
Apart from passing commands to the kernel, the main task of a shell is providing a user environment, which can be configured individually using shell resource configuration file
Shell types-
Just like people know different languages and dialects, your UNIX system will usually offer a variety of shell types:
➢ sh or Bourne Shell : the original shell still used on UNIX systems and in UNIX-related environments. This is the basic shell, a small program with few features. While this is not the standard shell, it is still available on every Linux system for compatibility with UNIX programs.
➢ bash or Bourne Again shell : the standard GNU shell, intuitive and flexible. Probably most advisable for beginning users while being at the same time a powerful tool for the advanced and professional user. On Linux, bash is the standard shell for common users. This shell is a so-called superset of the Bourne shell, a set of add-ons and plug-ins. This means that the Bourne Again shell is compatible with the Bourne shell: commands that work in sh, also work in bash. However, the reverse is not always the case. All examples and exercises in this book use bash.
➢ csh or C shell : the syntax of this shell resembles that of the C programming language. Sometimes asked for by programmers.
➢ tcsh or TENEX C shell : a superset of the common C shell, enhancing user-friendliness and speed. That is why some also call it the Turbo C shell.
➢ ksh or the Korn shell : sometimes appreciated by people with a UNIX background. A superset of the Bourne shell; with standard configuration a nightmare for beginning users.
The file /etc/shells gives an overview of known shells on a Linux system:
[oracle@OEL-11g ~]$ cat /etc/shells
/bin/sh
/bin/bash
/sbin/nologin
/bin/dash
/bin/tcsh
/bin/csh
/bin/ksh
Your default shell is set in the /etc/passwd file, like this line for user
[oracle@OEL-11g ~]$ echo $SHELL
/bin/bash
[oracle@OEL-11g ~]$ cat /etc/passwd | grep oracle
oracle:x:500:500::/home/oracle:/bin/bash
To switch from one shell to another, just enter the name of the new shell in the active terminal. The system finds the directory where the name occurs using the PATH settings, and since a shell is an executable file (program), the current shell activates it and it gets executed. A new prompt is usually shown, because each shell has its typical appearance:
[oracle@OEL-11g ~]$ echo $SHELL
/bin/bash
[oracle@OEL-11g ~]$ ksh
$ date
Sun Apr 26 21:34:28 IST 2015
$ exit
[oracle@OEL-11g ~]$ usermod oracle -s ksh
/usr/sbin/usermod: Permission denied.
[oracle@OEL-11g ~]$ cat /etc/passwd | grep oracle
oracle:x:500:500::/home/oracle:/bin/bash
Advantages of the Bourne Again Shell
➢ Interactive shells
➢ Non-Interactive shells
➢ Conditionals
➢ Shell arithmetic
➢ Aliases
➢ Arrays
➢ Directory stack
➢ The prompt
➢ The restricted shell
Activity:
o Check all the shells supported by linux.
cat /etc/shells
o Check the default shell in linux.
echo $SHELL
cat /etc/passwd | grep oracle
date
o Move from one shell to different shell.
Ksh
date
1.2. Executing commands
General-
Bash determines the type of program that is to be executed. Normal programs are system commands that exist in compiled form on your system. When such a program is executed, a new process is created because Bash makes an exact copy of itself. This child process has the same environment as its parent, only the process ID number is different. This procedure is called forking.
After the forking process, the address space of the child process is overwritten with the new process data. This is done through an exec call to the system.
The fork-and-exec mechanism thus switches an old command with a new, while the environment in which the new program is executed remains the same, including configuration of input and output devices, environment variables and priority. This mechanism is used to create all UNIX processes, so it also applies to the Linux operating system. Even the first process, init, with process ID 1, is forked during the boot procedure in the so-called bootstrapping procedure.
Shell built-in commands
Built-in commands are contained within the shell itself. When the name of a built-in command is used as the first word of a simple command, the shell executes the command directly, without creating a new process. Built-in commands are necessary to implement functionality impossible or inconvenient to obtain with separate utilities.
Bash supports 3 types of built-in commands:
Bourne Shell built-ins:
:, ., break, cd, continue, eval, exec, exit, export, getopts, hash, pwd, readonly, return, set, shift, test, [, times, trap, umask and unset.
Bash built-in commands:
alias, bind, builtin, command, declare, echo, enable, help, let, local, logout, printf, read, shopt, type, typeset, ulimit and unalias.
Special built-in commands:
When Bash is executing in POSIX mode, the special built-ins differ from other built-in commands in three respects:
Special built-ins are found before shell functions during command lookup.
If a special built-in returns an error status, a non-interactive shell exits.
Assignment statements preceding the command stay in effect in the shell environment after the command completes.
The POSIX special built-ins are :, ., break, continue, eval, exec, exit, export, readonly, return, set, shift, trap and unset.
Most of these built-ins will be discussed in the next chapters. For those commands for which this is not the case, we refer to the Info pages.
Executing programs from a script
She-Bang – #! : this will decide in which shell the script has to be executed. It should always be defined in the first line of the script.
When the program being executed is a shell script, bash will create a new bash process using a fork. This subshell reads the lines from the shell script one line at a time. Commands on each line are read, interpreted and executed as if they would have come directly from the keyboard.
While the subshell processes each line of the script, the parent shell waits for its child process to finish. When there are no more lines in the shell script to read, the subshell terminates. The parent shell awakes and displays a new prompt.
1.3. Building blocks
Shell building blocks:
Shell syntax
If input is not commented, the shell reads it and divides it into words and operators, employing quoting rules to define the meaning of each character of input. Then these words and operators are translated into commands and other constructs, which return an exit status available for inspection or processing. The above fork-and-exec scheme is only applied after the shell has analyzed input in the following way:
➢ The shell reads its input from a file, from a string or from the user’s terminal.
➢ Input is broken up into words and operators, obeying the quoting rules, see Chapter 3. These tokens are separated by meta characters. Alias expansion is performed.
➢ The shell parses (analyzes and substitutes) the tokens into simple and compound commands.
➢ Bash performs various shell expansions, breaking the expanded tokens into lists of filenames and commands and arguments.
➢ Redirection is performed if necessary, redirection operators and their operands are removed from the argument list
➢ Commands are executed.
➢ Optionally the shell waits for the command to complete and collects its exit status.
Shell commands
A simple shell command such as touch file1 file2 file3 consists of the command itself followed by arguments, separated by spaces.
More complex shell commands are composed of simple commands arranged together in a variety of ways: in a pipeline in which the output of one command becomes the input of a second, in a loop or conditional construct, or in some other grouping. A couple of examples:
ls | more
gunzip file.tar.gz | tar xvf –
Shell functions
Shell functions are a way to group commands for later execution using a single name for the group. They are executed just like a “regular” command. When the name of a shell function is used as a simple command name, the list of commands associated with that function name is executed.
Shell functions are executed in the current shell context; no new process is created to interpret them.
Shell parameters
A parameter is an entity that stores values. It can be a name, a number or a special value. For the shell’s purpose, a variable is a parameter that stores a name. A variable has a value and zero or more attributes. Variables are created with the declare shell built-in command.
If no value is given, a variable is assigned the null string. Variables can only be removed with the unset built-in.
Shell expansions
Shell expansion is performed after each command line has been split into tokens. These are the expansions performed:
➢ Brace expansion
➢ Tilde expansion
➢ Parameter and variable expansion
➢ Command substitution
➢ Arithmetic expansion
➢ Word splitting
➢ Filename expansion
Redirections
Before a command is executed, its input and output may be redirected using a special notation interpreted by the shell. Redirection may also be used to open and close files for the current shell execution environment.
Executing commands
When executing a command, the words that the parser has marked as variable assignments (preceding the command name) and redirections are saved for later reference. Words that are not variable assignments or redirections are expanded; the first remaining word after expansion is taken to be the name of the command and the rest are arguments to that command. Then redirections are performed, then strings assigned to variables are expanded. If no command name results, variables will affect the current shell environment.
An important part of the tasks of the shell is to search for commands. Bash does this as follows:
Check whether the command contains slashes. If not, first check with the function list to see if it contains a command by the name we are looking for.
If command is not a function, check for it in the built-in list.
If command is neither a function nor a built-in, look for it analyzing the directories listed in PATH. Bash uses a hash table (data storage area in memory) to remember the full path names of executables so extensive PATH searches can be avoided.
If the search is unsuccessful, bash prints an error message and returns an exit status of 127.
If the search was successful or if the command contains slashes, the shell executes the command in a separate execution environment.
If execution fails because the file is not executable and not a directory, it is assumed to be a shell script.
If the command was not begun asynchronously, the shell waits for the command to complete and collects its exit status.
Shell scripts
When a file containing shell commands is used as the first non-option argument when invoking Bash (without -c or -s, this will create a non-interactive shell. This shell first searches for the script file in the current directory, then looks in PATH if the file cannot be found there.
Activity:
o Check command is workable.
which cat
which cold
o Understand how a command gets executed on UNIX platform. Importance of PATH variable.
echo $PATH
o Write a command yourself and add location in path variable.
vi oracledate
#!/bin/bash
Date
chmod +x oracledate
oracledate
cp oracledate /oracle/base/product/12.1.0/db/bin
oracledate
1.4. Creating and running script
Writing and naming
A shell script is a sequence of commands for which you have a repeated use. This sequence is typically executed by entering the name of the script on the command line. Alternatively, you can use scripts to automate tasks using the cron facility. Another use for scripts is in the UNIX boot and shutdown procedure, where operation of daemons and services are defined in init scripts.
To create a shell script, open a new empty file in your editor. Any text editor will do: vim, emacs, gedit, dtpad et cetera are all valid. You might want to chose a more advanced editor like vim or emacs, however, because these can be configured to recognize shell and Bash syntax and can be a great help in preventing those errors that beginners frequently make, such as forgetting brackets and semi-colons.
Tip Syntax highlighting in vim
In order to activate syntax highlighting in vim, use the command
:syntax enable
or
:sy enable
or
:syn enable
You can add this setting to your .vimrc file to make it permanent.
Put UNIX commands in the new empty file, like you would enter them on the command line. As discussed in the previous chapter (see Section 1.3), commands can be shell functions, shell built-ins, UNIX commands and other scripts.
Give your script a sensible name that gives a hint about what the script does. Make sure that your script name does not conflict with existing commands. In order to ensure that no confusion can rise, script names often end in .sh; even so, there might be other scripts on your system with the same name as the one you chose. Check using which, whereis and other commands for finding information about programs and files:
which -a script_name
whereis script_name
locate script_name
script1.sh :
In this example we use the echo Bash built-in to inform the user about what is going to happen, before the task that will create the output is executed. It is strongly advised to inform users about what a script is doing, in order to prevent them from becoming nervous because the script is not doing anything. We will return to the subject of notifying users.
Write this script for yourself as well. It might be a good idea to create a directory ~/scripts to hold your scripts. Add the directory to the contents of the PATH variable:
export PATH=”$PATH:~/scripts”
If you are just getting started with Bash, use a text editor that uses different colours for different shell constructs. Syntax highlighting is supported by vim, gvim, (x)emacs, kwrite and many other editors; check the documentation of your favorite editor.
Note Different prompts
The prompts throughout this course vary depending on the author’s mood. This resembles much more real life situations than the standard educational $ prompt. The only convention we stick to, is that the root prompt ends in a hash mark (#).
Executing the script
The script should have execute permissions for the correct owners in order to be runnable. When setting permissions, check that you really obtained the permissions that you want. When this is done, the script can run like any other command:
infra:~/scripts> chmod u+x script1.sh
infra:~/scripts> ls -l script1.sh
-rwxrw-r– 1 infra infra 456 Dec 24 17:11 script1.sh
infra:~> script1.sh
The script starts now.
Hi, infra!
I will now fetch you a list of connected users:
3:38pm up 18 days, 5:37, 4 users, load average: 0.12, 0.22, 0.15
USER TTY FROM LOGIN@ IDLE JCPU PCPU WHAT
root tty2 – Sat 2pm 4:25m 0.24s 0.05s -bash
infra :0 – Sat 2pm ? 0.00s ? –
infra pts/3 – Sat 2pm 3:33m 36.39s 36.39s BitchX infra ir
infra pts/2 – Sat 2pm 3:33m 0.13s 0.06s /usr/bin/screen
I’m setting two variables now.
This is a string: black
And this is a number: 9
I’m giving you back your prompt now.
infra:~/scripts> echo $COLOUR
infra:~/scripts> echo $VALUE
infra:~/scripts>
This is the most common way to execute a script. It is preferred to execute the script like this in a subshell. The variables, functions and aliases created in this subshell are only known to the particular bash session of that subshell. When that shell exits and the parent regains control, everything is cleaned up and all changes to the state of the shell made by the script, are forgotten.
If you did not put the scripts directory in your PATH, and . (the current directory) is not in the PATH either, you can activate the script like this:
./script_name.sh
A script can also explicitly be executed by a given shell, but generally we only do this if we want to obtain special behavior, such as checking if the script works with another shell or printing traces for debugging:
rbash script_name.sh
sh script_name.sh
bash -x script_name.sh
The specified shell will start as a subshell of your current shell and execute the script. This is done when you want the script to start up with specific options or under specific conditions which are not specified in the script.
If you don’t want to start a new shell but execute the script in the current shell, you source it:
source script_name.sh
Tip source = .
The Bash source built-in is a synonym for the Bourne shell . (dot) command.
The script does not need execute permission in this case. Commands are executed in the current shell context, so any changes made to your environment will be visible when the script finishes execution:
infra:~/scripts> source script1.sh
–output ommitted—
infra:~/scripts> echo $VALUE
9
infra:~/scripts>
Which shell will run the script?
When running a script in a subshell, you should define which shell should run the script. The shell type in which you wrote the script might not be the default on your system, so commands you entered might result in errors when executed by the wrong shell.
The first line of the script determines the shell to start. The first two characters of the first line should be #!, then follows the path to the shell that should interpret the commands that follow.
Blank lines are also considered to be lines, so don’t start your script with an empty line.
For the purpose of this course, all scripts will start with the line
#!/bin/bash
As noted before, this implies that the Bash executable can be found in /bin.
Adding comments
You should be aware of the fact that you might not be the only person reading your code. A lot of users and system administrators run scripts that were written by other people. If they want to see how you did it, comments are useful to enlighten the reader.
Comments also make your own life easier. Say that you had to read a lot of man pages in order to achieve a particular result with some command that you used in your script. You won’t remember how it worked if you need to change your script after a few weeks or months, unless you have commented what you did, how you did it and/or why you did it.
Take the script1.sh example and copy it to commented-script1.sh, which we edit so that the comments reflect what the script does. Everything the shell encounters after a hash mark on a line is ignored and only visible upon opening the shell script file:
#!/bin/bash
# This script clears the terminal, displays a greeting and gives information
# about currently connected users. The two example variables are set and displayed.
clear # clear terminal window
echo “The script starts now.”
echo “Hi, $USER!” # dollar sign is used to get content of variable
echo
echo “I will now fetch you a list of connected users:”
echo
w # show who is logged on and
echo # what they are doing
echo “I’m setting two variables now.”
COLOUR=”black” # set a local shell variable
VALUE=”9″ # set a local shell variable
echo “This is a string: $COLOUR” # display content of variable
echo “And this is a number: $VALUE” # display content of variable
echo
echo “I’m giving you back your prompt now.”
echo
In a decent script, the first lines are usually comment about what to expect. Then each big chunk of commands will be commented as needed for clarity’s sake. Linux init scripts, as an example, in your system’s init.d directory, are usually well commented since they have to be readable and editable by everyone running Linux.
Activity:
o Writing a script.
vi script1.sh
o Locating your scripts.
find . -name script*
locate script* – requires local database of linux to be updated
updatedb
locate script*
o Ways of executing the script.
chmod 755 script1.sh
chmod +x script1.sh
vi script1.sh
./script1.sh
sh script1.sh
ksh script1.sh
bash script1.sh
o Comment appropriately.
vi script1.sh #Finding the current system time stamp
o Use SHE-BANG in every script.
#!/bin/bash – this one should be existing.
./script1.sh
o Write a script and crontab it.
vi script2.sh
#!/bin/bash date > /home/oracle/scripting/script2.out
chmod +x script2.sh
date
crontab -l
1.5. Debugging scripts
Debugging on the entire script
When things don’t go according to plan, you need to determine what exactly causes the script to fail. Bash provides extensive debugging features. The most common is to start up the subshell with the -x option, which will run the entire script in debug mode. Traces of each command plus its arguments are printed to standard output after the commands have been expanded but before they are executed.
This is the commented-script1.sh script ran in debug mode. Note again that the added comments are not visible in the output of the script.
infra:~/scripts> bash -x script1.sh
+ clear
+ echo ‘The script starts now.’
The script starts now.
+ echo ‘Hi, infra!’
Hi, infra!
+ echo
+ echo ‘I will now fetch you a list of connected users:’
I will now fetch you a list of connected users:
+ echo
+ w
4:50pm up 18 days, 6:49, 4 users, load average: 0.58, 0.62, 0.40
USER TTY FROM LOGIN@ IDLE JCPU PCPU WHAT
root tty2 – Sat 2pm 5:36m 0.24s 0.05s -bash
infra :0 – Sat 2pm ? 0.00s ? –
infra pts/3 – Sat 2pm 43:13 36.82s 36.82s BitchX infra ir
infra pts/2 – Sat 2pm 43:13 0.13s 0.06s /usr/bin/screen
+ echo
+ echo ‘I’\”m setting two variables now.’
I’m setting two variables now.
+ COLOUR=black
+ VALUE=9
+ echo ‘This is a string: ‘
This is a string:
+ echo ‘And this is a number: ‘
And this is a number:
+ echo
+ echo ‘I’\”m giving you back your prompt now.’
I’m giving you back your prompt now.
+ echo
There is now a full-fledged debugger for Bash, available at SourceForge. These debugging features are available in most modern versions of Bash, starting from 3.x.
Debugging on part(s) of the script
Using the set Bash built-in you can run in normal mode those portions of the script of which you are sure they are without fault, and display debugging information only for troublesome zones. Say we are not sure what the w command will do in the example commented-script1.sh, then we could enclose it in the script like this:
set -x # activate debugging from here
w
set +x # stop debugging from here
Output then looks like this: infra: ~/scripts> script1.sh
The script starts now.
Hi, infra!
I will now fetch you a list of connected users:
+ w
5:00pm up 18 days, 7:00, 4 users, load average: 0.79, 0.39, 0.33
USER TTY FROM LOGIN@ IDLE JCPU PCPU WHAT
root tty2 – Sat 2pm 5:47m 0.24s 0.05s -bash
infra :0 – Sat 2pm ? 0.00s ? –
infra pts/3 – Sat 2pm 54:02 36.88s 36.88s BitchX infrake
infra pts/2 – Sat 2pm 54:02 0.13s 0.06s /usr/bin/screen
+ set +x
I’m setting two variables now.
This is a string:
And this is a number:
I’m giving you back your prompt now.
infra: ~/scripts>
You can switch debugging mode on and off as many times as you want within the same script.
The table below gives an overview of other useful Bash options:
Table 2-1. Overview of set debugging options
Short notation Long notation Result
set -f set -o noglob Disable file name generation using metacharacters (globbing).
set -v set -o verbose Prints shell input lines as they are read.
set -x set -o xtrace Print command traces before executing command.
The dash is used to activate a shell option and a plus to deactivate it. Don’t let this confuse you!
In the example below, we demonstrate these options on the command line:
infra:~/scripts> set -v
infra:~/scripts> ls
ls
commented-scripts.sh script1.sh
infra:~/scripts> set +v
set +v
infra:~/scripts> ls *
commented-scripts.sh script1.sh
infra:~/scripts> set -f
infra:~/scripts> ls *
ls: *: No such file or directory
infra:~/scripts> touch *
infra:~/scripts> ls
* commented-scripts.sh script1.sh
infra:~/scripts> rm *
infra:~/scripts> ls
commented-scripts.sh script1.sh
Alternatively, these modes can be specified in the script itself, by adding the desired options to the first line shell declaration. Options can be combined, as is usually the case with UNIX commands:
#!/bin/bash -xv
Once you found the buggy part of your script, you can add echo statements before each command of which you are unsure, so that you will see exactly where and why things don’t work. In the example commented-script1.sh script, it could be done like this, still assuming that the displaying of users gives us problems:
echo “debug message: now attempting to start w command”; w
In more advanced scripts, the echo can be inserted to display the content of variables at different stages in the script, so that flaws can be detected:
echo “Variable VARNAME is now set to $VARNAME.”
Activity:
o create issue in the script.
vi script1.sh
date
dummy
cat /etc/hosts
o Full script debugging ways.
bash -x script1.sh
vi script1.sh
set -x
./script1.sh
vi script1.sh
#!/bin/bash -x
./script1.sh
o Partial debugging ways.
vi script1.sh
set -x dummy
set +x
To continue with Tutorial 2, please click here