CompTIA Linux+ Certification: Automating Jobs

As you begin building shell scripts, you'll probably start to wonder how to run and control them on your Linux system. So far, we've only run commands and scripts directly from the command-line interface in real-time mode. This isn't the only way to run scripts in Linux.

There are quite a few other options available for running your shell scripts on Linux systems. This guide examines different ways you can use to get your scripts started. Also, sometimes you might run into the problem of a script that gets stuck in a loop and you need to figure out how to get it to stop without having to turn off your Linux system. This guide also examines the different ways you can control how and when your shell script runs on your system.

Key Topics:
Running Scripts in Background Mode
Running Scripts without a Console
Sending Signals
Job Control
Running Like Clockwork

Running Scripts in Background Mode
There are times when running a shell script directly from the command-line interface is inconvenient. Some scripts can take a long time to process, and you may not want to tie up the command-line interface waiting. While the script is running, you can't do anything else in your terminal session. Fortunately, there's a simple solution to that problem. The following sections describe how to run your scripts in background mode on your Linux system.

Running in the Background
Running a shell script in background mode is a fairly easy thing to do. To run a shell script in background mode from the command-line interface, just place an ampersand symbol after the command:
$ ./test1.sh &
[1] 19555
$ This is a test program
Loop #1
Loop #2
$ ls -l
total 8
-rwxr--r-- 1 rich rich 219 Feb 26 19:27 test1.sh
$ Loop #3

When you place the ampersand symbol after a command, it separates the command from the Bash shell and runs it as a separate background process on the system. The first thing that displays is the line
[1] 19555

The number in the square brackets is the job number the shell assigns to the background process. The shell assigns each process started a unique job number. The next number is the process ID (PID) the Linux system itself assigns to the process. So every process running in a shell has a unique job number, and every process running on the Linux system has a unique PID.
As soon as the system displays these items, a new command-line interface prompt appears. You are returned to the shell, and the command you executed runs safely in background mode.
At this point, you can enter new commands at the prompt (as shown in the example). However, while the background process is still running, it still uses your terminal monitor for output messages. You'll notice from the example that the output from the test1.sh script appears in the output intermixed with any other commands that are run from the shell.

When the background process finishes, it displays a message on the terminal:
[1]+ Done ./test1.sh
This shows the job number and the status of the job (Done), along with the command used to start the job.

Running Multiple Background Jobs
You can start any number of background jobs at the same time from the command-line prompt:
[1] 19582
$ This is the test1 program output
Test 1 Loop #1 output
$ ./test2.sh &
[2] 19597
$ This is the test2 program output
Test 2 Loop #1 output
$ ./test3.sh &
[3] 19612
$ This is the test3 program output
Test 3 Loop #1 output
Test 1 Loop #2 output
Test 2 Loop #2 output
Test 3 Loop #2 output

Each time you start a new job, the shell assigns it a new job number, and the Linux system assigns it a new PID.

You can see that all of the scripts are running by using the ps command:
$ ps au
USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
rich 19498 0.0 1.2 2688 1628 pts/0 S 11:38 0:00 -bash
rich 19582 0.0 0.9 2276 1180 pts/0 S 11:55 0:00 /bin/bash ./test3.sh
rich 9597 0.1 0.9 2276 1180 pts/0 S 11:55 0:00 /bin/bash ./test2.sh
rich 19612 0.1 0.9 2276 1180 pts/0 S 11:55 0:00 /bin/bash ./test1.sh
rich 19639 0.0 0.4 1564 552 pts/0 S 11:56 0:00 sleep 10
rich 19640 0.0 0.4 1564 552 pts/0 S 11:56 0:00 sleep 10
rich 19641 0.0 0.4 1564 552 pts/0 S 11:56 0:00 sleep 10
rich 19642 0.0 0.5 2588 744 pts/0 R 11:56 0:00 ps au

Each of the background processes you start appears in the ps command output listing of running processes. If all of the processes display output in your terminal session, things can get pretty messy pretty quickly. Fortunately, there's a simple way to solve that problem, which we'll discuss in the next section.
ps command that each of the background processes is tied to the terminal session (pts/0) terminal. If the terminal session exits, the background process also exits. Some terminal emulators warn you if you have any running background processes associated with the terminal, while others don't. If you want your script to continue running in background mode after you've logged off the console, there's something else you need to do. The next section discusses that process.

Running Scripts without a Console
There will be times when you want to start a shell script from a terminal session and then let the script run in background mode until it finishes, even if you exit the terminal session. You can do this by using the nohup command.
The nohup command runs another command blocking any SIGHUP signals that are sent to the process. This prevents the process from exiting when you exit your terminal session (nohup is short for “no hangup”).

You can combine the nohup command with the ampersand to run a script in the background and not allow it to be interrupted:
$ nohup ./test1.sh &
[1] 19831
$ nohup: appending output to 'nohup.out'
Just as with a normal background process, the shell assigns the command a job number, and the Linux system assigns a PID number. The difference is that when you use the nohup command, the script ignores any SIGHUP signals sent by the terminal session if you close the session.

Because the nohup command disassociates the process from the terminal, the process loses the output link to your monitor. To accommodate any output generated by the command, the nohup command automatically redirects output messages to a file, called nohup.out, in the current working directory.

The nohup.out file contains all of the output that would normally be sent to the terminal monitor.

After the process finishes running, you can view the nohup.out file for the output results:
$ cat nohup.out
This is a test program
Loop #3
Loop #4
Loop #5
Loop #6
Loop #7
Loop #8
Loop #9
Loop #10
This is the end of the test program

The output appears in the nohup.out file just as if the process ran on the command line!
nohup, the output is appended to the existing nohup.out file. Be careful when running multiple commands from the same directory, as all of the output will be sent to the same nohup.out file, which can get confusing.

Sending Signals
As discussed in, “Optimizing Performance,” the Bash shell can send signals to processes running on the system. This allows you to stop or interrupt a runaway application process if necessary. While the kill and pkill commands discussed in this guide are good for stopping background processes, applications running in the foreground on the console are harder to control. Fortunately, there are two basic Linux signals you can generate using key combinations on the keyboard to interrupt or stop a foreground process.

Interrupting a Process
The Ctrl+C key combination generates a signal interrupt (SIGINT) signal and sends it to any processes currently running in the shell. You can test this by running a command that normally takes a long time to finish and pressing the Ctrl+C key combination:

$ sleep 100
The Ctrl+C key combination doesn't produce any output on the monitor; it just stops the current process running in the shell.

Pausing a Process
Instead of terminating a process, you can pause it in the middle of whatever it's doing. Sometimes this can be a dangerous thing (for example, if a script has a file lock open on a crucial system file), but often it allows you to peek inside what a script is doing without actually terminating the process.
The Ctrl+Z key combination generates a signal terminal stop (SIGTSTP) signal, stopping any processes running in the shell. Stopping a process is different than terminating the process, as stopping the process leaves the program still in memory and able to continue running from where it left off. In the following section, “Job Control,” you'll learn how to restart a process that's been stopped.

When you use the Ctrl+Z key combination, the shell informs you that the process has been stopped:
[1]+ Stopped sleep 100

The number in the square brackets indicates the job number for the process in the shell. If you have a stopped job assigned to your shell session, Bash will warn you if you try to exit the shell:
$ exit
logout
There are stopped jobs.

You can view the stopped job by using the ps command:
USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
rich 20560 0.0 1.2 2688 1624 pts/0 S 05:15 0:00 -bash
rich 20605 0.2 0.4 1564 552 pts/0 T 05:22 0:00 sleep 100
rich 20606 0.0 0.5 2584 740 pts/0 R 05:22 0:00 ps au

The ps command shows the status of the stopped job as T, which indicates the command either is being traced or is stopped. The original Bash shell is shown as S, indicating that it's sleeping, waiting for the script to end. The ps command itself is shown with an R status, indicating that it's the currently running job.
If you really want to exit the shell with the stopped job still active, just type the exit command again. The shell will exit, terminating the stopped job.

Alternately, now that you know the PID of the stopped job, you can use the kill command to send a signal kill (SIGKILL) signal to terminate it:
$ kill -9 20605
$
[1]+ Killed sleep 100

When you kill the job, initially you won't get any response. However, the next time you do something that produces a shell prompt, you'll see a message indicating that the job was killed. Each time the shell produces a prompt, it also displays the status of any jobs that have changed states in the shell.

Job Control
In the previous section you saw how to use the Ctrl+Z key combination to stop a job running in the shell. After you stop a job, the Linux system lets you either kill or restart it. Restarting a stopped process requires sending it a signal continue (SIGCONT) signal.
 

The function of starting, stopping, killing, and resuming jobs is called job control. With job control, you have full control over how processes run in your shell environment.
The following sections describe the commands you can use to view and control jobs running in your shell.

Viewing Jobs
The key command for job control is the jobs command. The jobs command allows you to view the current jobs being handled by the shell.

List: Stopping a running job
$ cat test2.sh
#!/bin/bash
# testing job control
echo "This is a test program $$"
count=1
while [ $count -le 10 ] ; do
echo "Loop #$count"
sleep 10
count=$[ $count + 1 ]
done
echo "This is the end of the test program"
$ ./test2.sh
This is a test program 29011
[1]+ Stopped ./test2.sh
$ ./test2.sh > test2.sh.out &
[2] 28861
$ jobs
[2]- Running ./test2/sh>test2.shout &

The script shown uses the $$ variable to display the PID that the Linux system assigns to the script; then it goes into a loop, sleeping for 10 seconds at a time for each iteration. In the example, we start the first script from the command-line interface and then stop it using the Ctrl+Z key combination. Next, another job is started as a background process, using the ampersand symbol. To make life a little easier, we redirected the output of that script to a file so that it wouldn't appear on the monitor.

After the two jobs were started, we used the jobs command to view the jobs assigned to the shell. The jobs command shows both the stopped and the running jobs along with their job numbers and the commands used in the jobs.
The jobs command uses a few different command-line parameters, shown below.

TABLE:  The jobs command parameters
 

You probably noticed the plus and minus signs in the output. The job with the plus sign is considered the default job. It would be the job referenced by any job control commands if a job number wasn't specified in the command line. The job with the minus sign is the job that would become the default job when the current default job finishes processing. There will only be one job with the plus sign and one job with the minus sign at any time, no matter how many jobs are running in the shell.

List: Demonstrating job control (an example of how the next job in line takes over the default status when the default job is removed.)
$ ./test2.sh
This is a test program 29075
This is a test program 29090
[2]+ Stopped ./test2.sh
This is a test program 29105
[3]+ Stopped ./test2.sh
$ jobs -l
[1] 29075 Stopped ./test2.sh
[2]- 29090 Stopped ./test2.sh
[3]+ 29105 Stopped ./test2.sh
$ kill -9 29105
[1]- 29075 Stopped ./test2.sh
[2]+ 29090 Stopped ./test2.sh

In the above list, we started, then stopped, three separate processes. The jobs command listing shows the three processes and their status. Note by the PID numbers that the default process (the one listed with the plus sign) is 29105, the last process started.

We then used the kill command to send a SIGHUP signal to the default process. In the next jobs listing, the job that previously had the minus sign, 29090, is now the default job.

Restarting Stopped Jobs
Under Bash job control, you can restart any stopped job as either a background process or a foreground process. A foreground process takes over control of the terminal you're working on, so be careful about using that feature.

To restart a job in background mode, use the bg command along with the job number:
$ bg 2

[2]+ ./test2.sh &
$ Test 2 Loop #3 output
Test 2 Loop #4 output
[2]- Running ./test2.sh &
$ Test 2 Loop #6 output
Test 2 Loop #7 output
Test 2 Loop #8 output
Test 2 Loop #9 output
Test 2 Loop #10 output
This is the end of the test2 program
[2]- Done ./test2.sh

Since we restarted the job in background mode, the command-line interface prompt appears, allowing us to continue with other commands. The output from the jobs command now shows that the job is indeed running (as you can tell from the output now appearing on the monitor).

To restart a job in foreground mode, use the fg command along with the job number:
$ jobs
$ fg 1
./test4
Loop #3

Since the job is running in foreground mode, we don't get a new command-line interface prompt until the job finishes.

Running Like Clockwork
We're sure that, as you start working with scripts, there will be a situation in which you'll want to run a script at a preset time, usually at a time when you're not there. There are two common ways of running a script at a preselected time:
The at command
The cron table

Each method uses a different technique for scheduling when and how often to run scripts. The following sections describe each of these methods.

Scheduling a Job Using the at Command
The at command allows you to specify a time when the Linux system will run a script. It submits a job to a queue with directions on when the shell should run the job. Another command, atd, runs in the background and checks the job queue for jobs to run. Most Linux distributions start this automatically at boot time.
The atd command checks a special directory on the system (usually /var/spool/at) for jobs submitted using the at command. By default, the atd command checks this directory every 60 seconds. When a job is present, the atd command checks the time the job is set to be run. If the time matches the current time, the atd command runs the job.

The following sections describe how to use the at command to submit jobs to run and how to manage jobs.

The at Command Format
The basic at command format is pretty simple:
at [-f filename] time

By default, the at command submits input from STDIN to the queue. You can specify a filename used to read commands (your script file) using the -f parameter.
The time parameter specifies when you want the Linux system to run the job. You can get pretty creative with how you specify the time. The at command recognizes lots of different time formats.

For example:
A standard hour and minute, such as 10:15
An AM/PM indicator, such as 10:15 p.m.

A specific named time, such as now, noon, midnight, or teatime (4 p.m.)

If you specify a time that's already past, the at command runs the job at that time on the next day.

Besides specifying the time to run the job, you can also include a specific date, using a few different date formats:
A standard date format, such as MMDDYY, MM/DD/YY, or DD.MM.YY
A text date, such as Jul 4 or Dec 25, with or without the year
You can also specify a time increment in different formats:
Now + 25 minutes
10:15 p.m. tomorrow
22:15 tomorrow
10:15 + 7 days

When you use the at command, the job is submitted to a job queue. The job queue holds the jobs submitted by the at command for processing. There are 26 different job queues available for different priority levels. Job queues are referenced using lowercase letters, a through z.
By default all at jobs are submitted to job queue a, the highest-priority queue.

If you want to run a job at a lower priority, you can specify the letter using the -q parameter.

Retrieving Job Output
When the job runs on the Linux system, there's no monitor associated with the job. Instead, the Linux system uses the email address of the user who submitted the job. Any output destined to STDOUT or STDERR is mailed to the user via the mail system.


List: Using the at command to start a job
Thu Feb 24 18:48:20 EST 2022
$ at -f test3.sh 18:49
job 2 at Thu Feb 24 18:49:00 2022
$ mail
Heirloom Mail version 12.5 7/5/10. Type ? for help.
"/var/spool/mail/rich": 1 message 1 new
>N 1 Rich Thu Feb 24 18:49 15/568 "Output from your job "
&
Message 1:
From [email protected] Thu Feb 24 18:49:00 2022
Return-Path: <[email protected]>
X-Original-To: rich
Delivered-To: [email protected]
Subject: Output from your job 2
To: [email protected]
Date: Thu, 24 Feb 2022 18:49:00 -0500 (EST)
From: [email protected] (Rich)
Status: R
"This script ran at 18:49:00"
"This is the end of the script"
&

As shown when we ran the at command, it produced a warning message, indicating what shell the system uses to run the script (the default shell assigned to /bin/sh, which for Linux is the Bash shell) along with the job number assigned to the job and the time the job is scheduled to run.
When the job completes, nothing appears on the monitor, but the system generates an email message. The email message shows the output generated by the script. If the script doesn't produce any output, it won't generate an email message, by default. You can change that by using the -m option in the at command. This generates an email message, indicating that the job completed, even if the script doesn't generate any output.

List: Jobs
The atq command allows you to view what jobs are pending on the system:
$ at -f test3.sh 10:15
warning: commands will be executed using /bin/sh
job 7 at 2022-03-04 10:15
$ at -f test5 4PM
job 8 at 2022-03-03 16:00
$ at -f test5 1PM tomorrow
job 9 at 2022-03-04 13:00
$ atq
7 2022-03-04 10:15 a
8 2022-03-03 16:00 a
9 2022-03-04 13:00 a

The job listing shows the job number, the date and time the system will run the job, and the job queue the job is stored in.

Removing Jobs
After you know the information about what jobs are pending in the job queues, you can use the atrm command to remove a pending job:
$ atrm 8
Just specify the number of the job you want to remove. You can only remove jobs that you submit for execution. You can't remove jobs submitted by others.

Scheduling Regular Scripts
Using the at command to schedule a script to run at a preset time is great, but what if you need that script to run at the same time every day or once a week or once a month? Instead of having to continually submit at jobs, you can use another feature of the Linux system.
The Linux system uses the cron program to allow you to schedule jobs that need to run on a regular basis. The cron program runs in the background and checks special tables, called cron tables, for jobs that are scheduled to run.

The Cron Table
The cron table uses a special format for allowing you to specify when a job should be run. The format for the cron table entry is as follows:
min hour dayofmonth month dayofweek command
The cron table allows you to specify entries as specific values, ranges of values (such as 1–5) or as a wildcard character (the asterisk). For example, if you want to run a command at 10:15 a.m. every day, you would use the cron table entry of
15 10 * * * command

Because you can't indicate a.m. or p.m. in the cron table, you'll need to use the 24-hour clock format for p.m. times. The wildcard character used in the dayofmonth, month, and dayofweek fields indicates that cron will execute the command every day of every month at 10:15 a.m.

To specify a command to run at 4:15 p.m. every Monday, you would use
15 16 * * 1 command

You can specify the dayofweek entry either as a three-character text value (mon, tue, wed, thu, fri, sat, sun) or as a numeric value, with 0 being Sunday and 6 being Saturday.

Here's another example: to execute a command at 12 noon on the first day of every month, you'd use the format
00 12 1 * * command
The dayofmonth entry specifies a date value (1–31) for the month.

When specifying the command or shell to run, you must use its full pathname. You can add any command-line parameters or redirection symbols you like, as a regular command line:
15 10 * * * /home/rich/test4.sh> test4out

The cron program runs the script using the user account that submitted the job. Thus, you must have the proper permissions to access the command and output files specified in the command listing.

Building the Cron Table
All system users can have their own cron table (including the root user) for running scheduled jobs. Linux provides the crontab command for handling the cron table.

To list an existing cron table, use the -l parameter:
$ crontab -l
no crontab for rich

To add entries to your cron table, use the -e parameter. When you do that, the crontab command automatically starts the vi editor with the existing cron table or an empty file if it doesn't yet exist.

Working with Systemd Timers
Systems that utilize the systemd startup method (read“Maintaining System Startup and Services”) can also use the Systemd timer feature to automatically start programs. The timer unit files allow you to define events that occur at specific dates or times, similar to how the cron program works. The timer unit files, though, allow you to fine-tune exactly when a program starts.
Timer unit files are designated by a .timer file extension and include a [Timer] section that defines the directives required to determine when to start the event.

The table below describes these directives.
As you can see below, timer units provide several options for how to set the timer that aren't available in the cron program, such as the amount of time since the program last completed. This allows you to choose exactly when a program should start on the system.
The following exercise walks you through working with jobs on your Linux system.

TABLE: Commonly used timer unit file [Timer] section directives

EXERCISE: Manipulating Jobs from the Command Line
This exercise walks you through how to run, pause, stop, and view jobs running within the Bash shell.

At the command prompt, open a text editor of your choice and create the text file jobtest.sh by typing nano jobtest.sh, pico jobtest.sh, or vi jobtest.sh.

Enter the following code in the new text file:
#!/bin/bash
# jobtest.sh - run the sleep command in a loop for job testing
echo "This is a test program $$"
count=1
while [ $count -le 10 ] ; do
echo "Program: $$ Loop #$count"
echo "This is the end of the test program"

Give yourself execute permissions to the file by typing chmod u+x jobtest.sh.
Run the shell script by typing ./jobtest.sh.
Pause the job by pressing Ctrl+Z.
Start another copy of the job in background mode by typing ./jobtest.sh &.
List: current shell jobs by typing jobs.
Restart the paused job in background mode by typing bg n, where n is the job number assigned to the paused job.
List: current shell jobs by typing jobs. Note the status of the job that was previously paused.
Stop the running background jobs by using the kill -9 command along with the appropriate PID values assigned to each job.
Type jobs to view the currently running jobs.

By default, when you run a script in a terminal session shell, the interactive shell is suspended until the script completes. You can cause a script or command to run in background mode by adding an ampersand sign (&) after the command name. When you run a script or command in background mode, the interactive shell returns, allowing you to continue entering more commands. Any background processes run using this method are still tied to the terminal session. If you exit the terminal session, the background processes also exit.

To prevent this from happening, use the nohup command. This command intercepts any signals intended for the command that would stop it, such as, for example, when you exit the terminal session. This allows scripts to continue running in background mode even if you exit the terminal session.

When you move a process to background mode, you can still control what happens to it. The jobs command allows you to view processes started from the shell session. Once you know the job ID of a background process, you can use the kill command to send Linux signals to the process or use the fg command to bring the process back to the foreground in the shell session. You can suspend a running foreground process by using the Ctrl+Z key combination and then place it back in background mode using the bg command.

Besides controlling processes while they're running, you can also determine when a process starts on the system. Instead of running a script directly from the command-line interface prompt, you can schedule the process to run at an alternative time. There are several different ways to accomplish this. The at command allows you to run a script once at a preset time. The cron program provides an interface that can run scripts at a regularly scheduled interval.

Important exam questions:

Describe how to run a shell script in background mode from your console or terminal session.
- To run a shell script in background mode, include the ampersand sign (&) after the shell script command on the command line. The shell will run the script in background mode and produce another command prompt for you to continue within the shell.

Explain how to disconnect a shell script from the console or terminal session so that it continues running if the session closes.
- The nohup command disconnects the shell script from the shell session and runs it as a separate process. If the console or terminal session exits, the shell script will continue running.

Explain how to stop or pause a shell script running in the foreground on a console or terminal session.
- To stop a shell script running in the foreground of a console or terminal session, press the Ctrl+C key combination. To pause a running shell script, press Ctrl+Z.

Describe how to list shell scripts running in background mode within a console or terminal session.
- The jobs command allows you to list the commands that are running within the console or terminal session. The output from the jobs command displays both the job number assigned by the shell and the process ID assigned by the Linux system.

Describe how to run a shell script at a specific time.
- The at command allows you to schedule a job to run at a specific time. You can specify the time by using an exact value, such as 10:00 p.m., or by using common date and time references, such as 10:00 a.m. tomorrow.

Explain how to run a shell script automatically at a set time every day.
- The cron process runs every minute and checks for jobs that are scheduled to run. You must define the jobs to run in the cron table by using the crontab command.