CompTIA Linux+
Random

Click random to get a fresh chapter.

CompTIA Linux+ Certification: Tending Kernel Modules

Objective 1.7: Given a scenario, manage software configurations.
A module (also called a kernel module) is a self-contained driver library file. The advantage of using modules, instead of compiling all their features into the kernel, is that they keep the Linux kernel lighter and more agile. We can add certain kernel functionality as needed or on demand because modules can be loaded and unloaded into the kernel dynamically. They extend the functionality of the kernel without the need to reboot the system.
We'll take a look at the various kernel module types, where their files are stored, and module configuration file locations. We'll also explore in this guide how to dynamically link and unlink the modules, view module information, and remove modules.

Key Topics:
Exploring Kernel Modules
Installing Kernel Modules
Removing Kernel Modules

Exploring Kernel Modules
Kernel modules come in different flavors. They are as follows:
Device driver: Facilitates communication with a hardware device.
Filesystem driver: Required for filesystem I/O.
Network driver: Used to implement network protocols.
System calls: Provides additional functions for adding/modifying system services.
Executable loader: Allows additional executable formats to load.

There are a few different files and directories you should be familiar with when working with modules. Modules required to support a kernel are stored in the /lib/modules/ directory tree. Each Linux kernel version available on your system has its own subdirectory within the /lib/modules/ directory. 

List: Viewing a /lib/modules directory
$ ls -F /lib/modules
5.11.0-40-generic/ 5.4.0-31-generic/ 5.4.0-47-generic/
5.4.0-26-generic/ 5.4.0-42-generic/ 5.4.0-48-generic/

Be aware that distributions may implement this directory a little differently. For instance, Red Hat–based distributions have the /lib/modules/ directory hard-linked to the /usr/lib/modules/ directory. Thus, they are the same directory, but with two different names. 

List: Viewing a /lib/modules and a /usr/lib/modules directory
4.18.0-305.19.1.el8_4.x86_64/ 4.18.0-348.2.1.el8_5.x86_64/
4.18.0-305.3.1.el8_4.x86_64/ 4.18.0-348.el8.x86_64/
$ ls -F /usr/lib/modules
$ ls -id /lib/modules
16808755 /lib/modules
$ ls -id /usr/lib/modules
16808755 /usr/lib/modules

Notice in the above list that the two directories share the same inode number. Hard links were originally covered in “Managing Files, Directories, and Text.”
If needed, you can customize a module to define any unique parameters required, such as hardware settings essential for the device to operate.

On some older Linux distributions there is a single configuration file, /etc/modules.conf, and on more modern distributions there are configuration directories:
/etc/modprobe.d/ or /etc/modules-load.d/ contains configuration files generated at system installation or created by an administrator.
/lib/modprobe.d/ stores configuration files generated by third-party software packages.
/usr/lib/modprobe.d/, if it exists, is typically hard-linked to the /lib/modprobe.d/directory.
/run/modprobe.d/ stores configuration files generated at runtime.

Within each configuration directory are multiple configuration files that have a .conf filename extension. 

List: Viewing /etc/modprobe directories
$ ls /etc/modprobe.d
firewalld-sysctls.conf lockd.conf nvdimm-security.conf tuned.conf
kvm.conf mlx4.conf truescale.conf vhost.conf
$ ls /lib/modprobe.d
dist-alsa.conf dist-blacklist.conf libmlx4.conf systemd.conf

In the Linux virtual directory system are many potential locations for kernel modules as well as module configuration files. Therefore, it is wise to take a look around your own Linux server and note their locations.

Note: systemd-modules-load.service handles loading kernel modules at boot time. You can find the various directories it may load modules from by using grep on the service unit file and searching for the ConditionDirectoryNotEmpty directive.
Many device driver kernel modules are loaded either at system boot time or dynamically when hardware devices are attached to the system. If problems occur, you need to know what utilities to use to help you diagnose the issue.

There are three handy programs that can help with modules:
dmesg displays the current kernel ring buffer.
lsmod shows brief module information.
modinfo provides detailed module data.

When it comes to modules, a module failure sometimes triggers a kernel message. The dmesg command is handy in that you can view kernel messages related to current events. At boot time, your distribution may take a snapshot of the kernel ring buffer and store the data in a file (typically the /var/log/dmesg file). Both of these ring buffer information sources can help you track down kernel module problems.
- A ring buffer is a fixed-size data structure in memory. It is not shaped in a ring but instead more like a tube. In the kernel ring buffer, as new messages enter the tube, the older messages are moved toward the structure's end and the oldest messages “drop out” of the tube's end (are deleted).
The dmesg utility will simply dump the current kernel ring buffer to STDOUT. It is helpful to employ the grep command to dig through the messages. 

List: Using dmesg with grep to display module messages
$ dmesg | grep -i driver
[ 1.674321] e1000: Intel(R) PRO/1000 Network Driver - version 7.3.21-k8-NAPI
[ 3.614747] cdrom: Uniform CD-ROM driver Revision: 3.20
[ 48.828793] tun: Universal TUN/TAP device driver, 1.6
[ 8760.969714] usbcore: registered new interface driver usb-storage

You can employ different search terms to filter the dmesg utility's output. For example, if you want to find information only on a USB device issue, you could pipe the dmesg output to the grep -i usb command.
The lsmod utility displays the status of modules currently within the Linux kernel.


List: Employing lsmod to display module status
$ lsmod
Module Size Used by
af_packet 49152 4
bridge 172032 1 ebtable_broute
stp 16384 1 bridge
btrfs 1327104 1
scsi_dh_alua 20480 0

In the lsmod output, each module is listed on a separate line. The first column is the module's name.
Notice in the above List the Used by column. The digit in this column indicates the number of processes or other modules currently using the module. If it is another kernel module using it, the other's module's name is displayed.

Note: lsmod utility displays by looking at the /proc/modules file's contents. However, it is not as nicely formatted.
You can find out more detailed information concerning a particular kernel module via the modinfo utility. It may require super user privileges.

List: Using modinfo to display detailed module information
$ sudo modinfo bridge
filename: /lib/modules/[...]/kernel/net/bridge/bridge.ko
alias: rtnl-link-bridge
version: 2.3
license: GPL
suserelease: openSUSE Leap 15.0
srcversion: D39BA7E56E769F636E31A8C
depends: stp,llc
retpoline: Y
intree: Y
vermagic: [...]SMP mod_unload modversions retpoline

Notice the kernel module's filename in the above List. Kernel module files typically have a .ko file extension. Also notice that the module's version number is displayed. This is helpful if you need to track down known bugs related to a particular module.
sysctl command, with the -a option:
$ sysctl -a
kernel.module_disable = 0

If the kernel.module_disable option is set to a value of 1, you won't be able to install modules. Most Linux distributions store kernel settings that are applied at boot time in the /etc/sysctl.conf file, or as separate configuration files in the /etc/sysctl.d directory. Check those files to see if the option is set, and edit the file if you need to add modules.

Installing Kernel Modules
Linux typically automatically loads modules as they are needed on the system. Often, though, you may want to test a new module or try new module configurations. To do so, you may need to manually install modules into the kernel. This is also called inserting or loading a module.

In this section, we'll look at a few utilities that can help you load modules into the kernel. They are as follows:
insmod
modprobe
depmod

The insmod utility allows you to insert a single module into the Linux kernel. Unfortunately, because it is so basic, you have to provide an absolute directory reference to the module file. Also, the command does not load any needed module dependencies. 

List: Using insmod to insert a single module into the kernel
$ lsmod | grep -i joydev
$ sudo insmod /lib/modules/[...]/kernel/drivers/input/joydev.ko
$ lsmod | grep -i joydev
joydev 24576 0

In the above list, the system is checked for a loaded module that has a module name of joydev using the lsmod command. It is not found. Thus, the insmod command inserts it into the kernel using its full filename and directory location. The lsmod command is employed again to show that the module is now indeed loaded into the kernel.

The modprobe command is easier to use than the insmod utility because you can denote modules by their module name. It also loads any additional modules that the inserted module needs to operate (dependencies).

A snipped example is shown in the List below which employs the -v switch on the modprobe command to display more information while it inserts a module and all of its currently unloaded dependencies.

List: Using modprobe to insert a module and its dependencies
$ sudo modprobe -v dm_mirror
insmod /lib/modules/[...]/kernel/drivers/md/dm-log.ko
insmod /lib/modules/[...]/kernel/drivers/md/dm-region-hash.ko
insmod /lib/modules/[...]/kernel/drivers/md/dm-mirror.ko
$ lsmod | grep -i dm_mirror
dm_mirror 28672 0
dm_region_hash 16384 1 dm_mirror
dm_log 16384 2 dm_mirror,dm_region_hash
dm_mod 139264 3 dm_mirror,dm_log,dm_multipath
$ sudo modinfo dm_mirror
filename: /lib/modules/[...]/kernel/drivers/md/dm-mirror.ko
author: Joe Thornber
description: device-mapper mirror target
srcversion: A784B0C071D49F47F94E83B
depends: dm-region-hash,dm-mod,dm-log
[...]SMP mod_unload modversions retpoline
raid1_resync_throttle:A percentage [...]

The dm_mirror module allows volume managers to mirror logical volumes.

In the above List, when the modprobe command is used to load this module, it loads two other modules as well, which the dm_mirror module needs to work properly. Notice that the modprobe utility is calling the insmod utility to perform the insertion work. Also notice that the dm_mirror module has a slightly different filename, dm-mirror.ko, shown in the modinfo utility's output.
The modprobe program uses the modules.dep file to determine any module dependencies. This file is typically located in the /lib/modules/ subdirectory.

List: Viewing the modules.dep dependencies file
$ ls /lib/modules/[...]/modules.dep
/lib/modules/[...]/modules.dep
$ grep -i mirror /lib/modules/[...]/modules.dep
kernel/drivers/md/dm-mirror.ko:
kernel/drivers/md/dm-region-hash.ko
kernel/drivers/md/dm-log.ko
kernel/drivers/md/dm-mod.ko

In the above list, the modules.dep file is searched for the word mirror using the grep utility. To see a particular module's dependencies, you locate the module's filename within the file. After the colon (:), the module's dependencies are listed by their full module filename. So for the dm_mirror modules (dm-mirror.ko), the module dependencies are the dm-region-hash, dm-log, and dm-mod modules. This corresponds with what was shown in the modinfo utility's output in the above List.
You can employ the depmod command to scan through the system looking for any hardware that was not automatically detected. This is useful for troubleshooting problems with new devices. 

List: Using the depmod utility to update the modules.dep file
$ sudo depmod -v
/lib/modules/[...]/kernel/sound/soc/intel/
atom/snd-soc-sst-atom-hifi2-platform.ko needs
"snd_pcm_lib_preallocate_pages_for_all":
/lib/modules/[...]/kernel/sound/core/snd-pcm.ko

In the above list the depmod utility scans the system, determines any needed modules, reviews the modules' dependencies, and updates the appropriate modules.dep file. Notice that it also displays its activity to STDOUT.

Removing Kernel Modules
It's a good idea to remove any kernel modules you are no longer using on your Linux system. If you just need to remove (unload) a module with no dependencies, you can employ the rmmod command. 

List: Using the rmmod utility to remove a module
$ lsmod | grep joydev
$ sudo rmmod -v joydev
$ lsmod | grep joydev

Notice in the above list that the rmmod utility understands module names, so you don't have to provide an absolute directory reference to the module file. Once the module is unloaded, the lsmod utility no longer displays the module's name in its output.

The modprobe utility is useful for removing modules that have one or more dependencies. You just need to add the -r switch, and if you desire detailed information, include the -v switch.

List: Using the modprobe utility to remove a module and its dependencies
$ sudo modprobe -rv dm_mirror
rmmod dm_mirror
rmmod dm_region_hash
rmmod dm_log

In the above List, the module dm_mirror is unloaded along with its two dependencies. Note that if the module was not loaded, you would not see any messages and just get a command-line prompt back.

When kernel modules fail or when you need to test a new module, it is vital to understand where module files as well as their configuration files are located. Equally important is the ability to diagnose problems using the various command-line utilities available for this purpose. Because modules can be dynamically linked and unlinked with the kernel, you should understand how to perform these tasks using the correct tools. These additional tools in your Linux tool belt will allow you to quickly resolve issues concerning kernel modules.

Important Exam Questions:

Describe the locations of kernel module files.- Kernel module files have a .ko file extension and are typically located in a subdirectory of the /lib/modules/ directory. There is a subdirectory for each particular Linux kernel version. Some distributions have additional directories, such as /usr/lib/modules/, which are hard-linked to the /lib/modules/ directory.

1. Distinguish the locations of module configuration files.
- Older Linux distributions use a single file, /etc/modules.conf, as their kernel modules configuration file. More modern distributions use configuration directories, which can be the /etc/modprobe.d/, /etc/modules-load.d/, /lib/modprobe.d/, /usr/lib/modprobe.d/, and/or /run/modprobe.d/ directory. Within configuration directories, module configuration files have a .conf file extension.

2. Summarize the utilities used to troubleshoot modules.
- Because when kernel modules fail they often issue a kernel message, you can employ the dmesg utility to view recent kernel messages or peruse the /var/log/dmesg file, if available, for boot time kernel problems. The lsmod utility displays all the currently loaded modules, the number of processes and other modules using them, and the other modules’ names. The modinfo program is very helpful because it displays detailed information concerning a module, including its dependencies.

3. Compare the utilities used to install kernel modules.
- The low-level insmod utility requires a full module filename in order to insert a module into the kernel, which can be cumbersome. In addition, it does not load any module dependencies. On the other hand, the modprobe utility only requires the module's name. Also, it searches the modules.dep file to determine and load any module dependencies.

4. Explain the utilities used to remove kernel modules.
- The rmmod utility is a low-level utility. Though it does not require a full module filename in order to unlink a module from the kernel, it does not unload any module dependencies. So you could end up with unneeded modules, still linked to the kernel. The modprobe utility, using the -r option, will unload the module and unlink any module dependencies.