CompTIA Linux+ Certification: Implementing Logging Services

Key Topics:
Understanding the Importance of Logging
Basic Logging Using rsyslog
Journaling with systemd-journald

Lots of things happen on a Linux system while it's running. Part of your job as a Linux administrator is knowing just what is happening and watching for when things go wrong. The primary tool for accomplishing that task is the logging service. All Linux distributions implement some type of logging service that tracks system events and stores them in log files. This guide explores the two most popular logging methods used in Linux distributions: rsyslog and systemd-journald.

First, this guide explains Linux logging principles to help give you an idea of what logging is all about. Then the guide discusses both the rsyslogd and the systemd-journald methods of generating logs.

Understanding the Importance of Logging
All Linux distributions implement some method of logging. Logging directs short messages that indicate what events happen, and when they happen, to users, files, or even remote hosts for storage. If something goes wrong, the Linux administrator can review the log entries to help determine the cause of the problem.
The following sections discuss the basics of how logging has been implemented in Linux and show the main logging packages that you'll most likely run into while working with various Linux distributions.

The syslog Protocol
In the early days of Unix, myriad logging methods were used to track system and application events. Applications used different logging methods, making it difficult for system administrators to troubleshoot issues.
In the mid-1980s, Eric Allman defined a protocol called syslog for logging events from his Sendmail mail application. The syslog protocol quickly became a standard for logging both system and application events in Unix and made its way to the Linux world.
What made the syslog protocol so popular is that it defines a standard message format that specifies the timestamp, type, severity, and details of an event. That standard can be used by the operating system, applications, and even devices that generate errors.
The type of event is defined as a facility value. The facility defines what is generating the event message, such as a system resource or an application.


TABLE: The syslog protocol facility values

As you can tell from the table, the syslog protocol covers many different types of events that can appear on a Linux system.
Each event is also marked with a severity. The severity value defines how important the message is to the health of the system.


TABLE: The syslog protocol severity values

Combining the facility and severity codes with a short informational message provides enough logging information to troubleshoot almost any problem in Linux.

The History of Linux Logging
Over the years there have been many open source logging projects for Linux systems. The following have been the most prominent:
Sysklogd: The original syslog application, this program includes two programs: the syslogd program to monitor the system and applications for events and the klogd program to monitor the Linux kernel for events.
Syslogd-ng: This program added advanced features, such as message filtering and the ability to send messages to remote hosts.
Rsyslog: The project claims the r stands for rocket fast. Speed is the focus of the rsyslog project, and the rsyslog application quickly became the standard logging package for many Linux distributions.
Systemd-journald: This is part of the Systemd application for system startup and initialization (see “Maintaining System Startup and Services”). Many Linux distributions are now using it for logging. It does not follow the syslog protocol but uses a completely different way of reporting and storing system and application events.

The following sections dive into the details of the two most popular logging applications: rsyslog and systemd-journald.

Basic Logging Using rsyslog
The rsyslog application utilizes all of the features of the original syslog protocol, including the configuration format and logging actions. The following sections walk you through how to configure the rsyslog logging application and where to find the common log files it generates.

Configuration
The rsyslog package uses the rsyslogd program to monitor events and log them as directed, using the /etc/rsyslog.conf configuration file to define what events to listen for and how to handle them. Many Linux distributions also use the /etc/rsyslog.d directory to store individual configuration files that are included as part of the rsyslog.conf configuration. This allows separate applications to define their own log settings.
The configuration file contains rules that define how the program handles syslog events received from the system, kernel, or applications. The format of an rsyslogd rule is as follows:
facility.priority action
The facility entry uses one of the standard syslog protocol facility keywords. The priority entry uses the severity keyword as defined in the syslog protocol, but with a twist. When you define a severity, syslogd will log all events with that severity or higher (lower severity code).

Thus, the entry
kern.crit
logs all kernel event messages with a severity of critical, alert, or emergency. To log only messages with a specific severity, use an equal sign before the priority keyword:
kern.=crit

You can also use wildcard characters for either the facility or priority. The entry
*.emerg
logs all events with an emergency severity level.

The action entry defines what rsyslogd should do with the received syslog message. The six action options you have available are as follows:
Forward to a regular file
Pipe the message to an application
Display the message on a terminal or the system console
Send the message to a remote host
Send the message to a list of users
Send the message to all logged-in users

List: The rsyslog.conf configuration entries for Ubuntu 20.04
auth,authpriv.* /var/log/auth.log
*.*;auth,authpriv.none -/var/log/syslog
kern.* -/var/log/kern.log
mail.* -/var/log/mail.log
mail.err /var/log/mail.err
*.emerg :omusrmsg:*

The first entry shown in the above list  defines a rule to handle all auth and authpriv facility type messages. This shows that you can specify multiple facility types by separating them with commas. The rule also uses a wildcard character for the priority, so all severity levels will be logged. This rule indicates that all security event messages will be logged to the /var/log/auth.log file.
The second entry defines a rule to handle all events (*.*) except security events (the .none priority). The event messages are sent to the /var/log/syslog file. The minus sign in front of the filename tells rsyslogd not to sync the file after each write, increasing the performance. The downside to this is if the system crashes before the next normal system sync, you may lose the event message.
The kern.* entry defines a rule to store all kernel event messages in a separate log file, located in the /var/log/kern.log file. This has become somewhat of a standard in Linux distributions.
The *.emerg entry defines a rule to handle all emergency events. The omusrmsg command indicates to send the event message to a user account on the system. By specifying the wildcard character, this rule sends all emergency event messages to all users on the system.
For comparison,


List: The rsyslog.conf configuration file for Rocky Linux 8.5
*.info;mail.none;authpriv.none;cron.none /var/log/messages
authpriv.* /var/log/secure
mail.* -/var/log/maillog
cron.* /var/log/cron
*.emerg :omusrmsg:*
uucp,news.crit /var/log/spooler
local7.* /var/log/boot.log

Notice that Red Hat–based systems use the /var/log/messages file for informational messages and the /var/log/secure file for security messages.

Note: logrotate utility. It automatically splits rsyslogd log files into archive files based on a time or the size of the file. You can usually identify archived log files by the numerical extension added to the log filename.

Making Log Entries
You may want to log your own application events. You can do that with the logger command-line tool:
logger [-isd] [-f file] [-p priority] [-t tag] [-u socket] [message]

The -i option specifies the process ID (PID) of the program that created the log entry as part of the event message. The -p option allows you to specify the event priority. The -t option allows you to specify a tag to add to the event message to help make finding the message in the log file easier. You can specify the message either as text in the command line or as a file using the -f option. The -d and -u options are advanced options for sending the event message to the network. The -s option sends the event message to the standard error output.

An example of using logger in a script would look like this:
$ logger This is a test message from rich

On an Ubuntu system, you can look at the end of the /var/log/syslog file to see the log entry:
$ tail /var/log/syslog
Feb 8 20:21:02 myhost rich: This is a test message from rich
Notice that rsyslogd added the timestamp, host, and user account for the message. This is a great troubleshooting tool!

Finding Event Messages
Generally, most Linux distributions create log files in the /var/log directory. Depending on the security of the Linux system, many log files are readable by everyone, but some may not be.

Most Linux distributions create separate log files for different event message types, although they don't always agree on the log filenames.
It's also common for individual applications to have a separate directory under the /var/log directory for their own application event messages, such as /var/log/apache2 for the Apache web server.

Note: /etc/rsyslog.conf configuration file. Just remember to also look for additional configuration files in the /etc/rsyslog.d directory.
Since rsyslogd log files are text files, you can use any of the standard text tools available in Linux, such as cat, head, tail, and, of course, vi to view them. One common trick for administrators is to use the -f option with the tail command. That displays the last few lines in the log file but then monitors the file for any new entries and displays those too.

Note: rsyslogd, some Linux distributions also include the lastb tool. It displays the event messages from the /var/log/wtmp log file, used by many Linux distributions to log user logins.

Journaling with systemd-journald
The Systemd system services package includes the systemd-journald journal utility for logging. Notice that we called it a journal utility instead of a logging utility. The systemd-journald program uses its own method of storing event messages, completely different from how the syslog protocol specifies.
The following sections discuss how to use the systemd-journald program to track event messages on your Linux system.

The systemd-journald service reads its configuration from the /etc/systemd/journald.conf configuration file. When you examine this file, you'll notice that there aren't any rules defined, only settings that control how the application works:
The Storage setting determines how systemd-journald stores event messages. When the setting is set to auto, it will look for the /var/log/journal directory and store event messages there. If that directory doesn't exist, it stores the event messages in the temporary /run/log/journal directory, which is deleted when the system shuts down. You must manually create the /var/log/journal directory for the event messages to be stored permanently. If you set the Storage setting to persistent, systemd-journald will create the directory automatically. When set to volatile, it only stores event messages in the temporary directory.
The Compress setting determines whether to compress the journal files.
There are several file maintenance settings that control how much space the journal is allowed to use as well as how often to split journal files for archive, based on either time or file size.
The ForwardToSyslog setting determines if systemd-journald should forward any received messages to a separate syslog program, such as rsyslogd, running on the system. This provides two layers of logging capabilities.
There are quite a few settings that allow you to customize exactly how systemd-journald works in your system. For a full list and explanation of all the settings, type man journald.conf at the command prompt.

Viewing Logs
The systemd-journald program doesn't store journal entries in text files. Instead it uses its own binary file format that works like a database. While this makes it a little harder to view journal entries, it does provide for quick searching for specific event entries.
The journalctl program is our interface to the journal files. The basic format for the journalctl command is as follows:
journalctl [options] [matches]
The options control data returned by the matches is displayed.


TABLE:  The journalctl command options

The matches parameter defines what type of journal entries to display.


TABLE: The journalctl matches parameter

The journalctl command is great for when you are looking for specific event entries in the journal; it allows you to filter out events using the matches and determine how to display them using the options.

List: Output from the journalctl command
[rich@localhost ~]$ journalctl -r _TRANSPORT=kernel
-- Logs begin at Fri 2019-02-08 12:47:04 EST, end at...
Feb 08 12:48:36 localhost.localdomain kernel: TCP: lp registered
Feb 08 12:48:17 localhost.localdomain kernel: fuse init (API version 7.22)
Feb 08 12:47:43 localhost.localdomain kernel: virbr0: port 1(virbr0-nic)
Feb 08 12:47:43 localhost.localdomain kernel: IPv6: ADDRCONF(NETDEV_UP)
Feb 08 12:47:35 localhost.localdomain kernel: e1000: enp0s8 NIC Link is Up
Feb 08 12:47:32 localhost.localdomain kernel: IPv6: ADDRCONF(NETDEV_CHANGE)
Feb 08 12:47:32 localhost.localdomain kernel: ip_set: protocol 6
Feb 08 12:47:32 localhost.localdomain kernel: IPv6: ADDRCONF(NETDEV_UP)
That makes digging through journal files much easier!

EXERCISE: Creating a Log or Journal Entry

This walks you through monitoring the system logs in real time and watching as you send a message to the logging facility on your Linux system.

This exercise demonstrates how to create a log or journal entry and view that entry in both rsyslogd logs and systemd-journald journals.
Log into your Linux graphical desktop and open two new command prompt windows. If you're using virtual terminals, open two separate virtual terminal sessions.
Start monitoring the standard log file in one command prompt window or virtual terminal session. On Ubuntu systems, use sudo tail -f /var/log/syslog. For Red Hat–based systems such as Fedora, Rocky, and CentOS, use sudo tail -f /var/log/messages.
In the second command prompt window or virtual terminal session, create a log event by using the command logger This is a test log entry.
Observe the output in the first window or virtual terminal session. You should see the new log entry appear at the end of the log file appropriate for your system.
Display the end of the journal by using the command journalctl -r. You should see your log event message appear in the journal as well. Both Debian and Red Hat–based systems use systemd-journald but also forward event messages to the rsyslogd application, so you can often see events in both systems.
Logging events that occur on your Linux system or applications is crucial to being able to troubleshoot problems. The Linux system has adopted the syslog protocol method for handling event messages. The syslog protocol defines an event type, called the facility, and an event severity. This helps you determine which events are important to act on and which ones are just informational.
The rsyslogd logging application utilizes the syslog protocol to log system and application events. The /etc/rsyslogd.conf configuration file controls what events are logged and how they are logged. The rsyslogd application can log events to files, applications, remote hosts, terminals, and even directly to users.
The Linux Systemd services package also includes its own method for logging events called journaling. The systemd-journald application monitors the system and applications for all events and sends them to a special journal file. The operation of systemd-journald is controlled by the /etc/systemd/journald configuration file. You must use the journalctl application to read events stored in the journal file.

Important Exam Questions

1. Describe the logging protocol used by most Linux logging applications.
- The syslog protocol has become the de facto standard for most Linux logging applications. It identifies events using a facility code, which defines the event type, and a severity, which defines how important the event message is. The sysklogd, syslogd-ng, and rsyslogd applications all use the syslog protocol for managing system and application events in Linux.

2. Describe how the rsyslogd application directs events to specific locations.
- The rsyslogd application uses the /etc/rsyslogd.conf configuration file to define rules for handling events. Each rule specifies a syslog facility and severity along with an action to take. Events that match the facility and have a priority equal to or higher than the severity defined are sent to the defined action. The action can be sending the event message to a log file, piping the message to an application, or sending the event message to a remote host or to a user on the system.

3. Explain how the Systemd service uses a different method for logging events.
- The Systemd service package uses the systemd-journald application, which doesn't use the syslog protocol for logging events. Instead, systemd-journald creates its own binary journal files for storing event messages. The binary journal file is indexed to provide faster searching for events. The journalctl application provides the interface for sending search queries to the journal files and displaying the search results.