CompTIA Server+ Certification: Troubleshooting (Hardware, Software, Storage, and Network Problems), and Performance Optimization

Objectives:
- Learn how to apply a troubleshooting methodology to real-world problems
- Troubleshoot hardware, software, storage, and network problems
- Apply troubleshooting skills to resolve security issues
- Optimize server and network performance

Topics:
Troubleshooting Methodology
Identify the Problem
Establish a Theory of Probable Cause
Test the Theory
Establish a Plan of Action
Implement a Solution or Escalate
Verify Functionality
Perform Root Cause Analysis
Document the Solution
Hardware Problems and Solutions
Software Problems and Solutions
Storage Problems and Solutions
Windows Tools
Linux Tools
Network Problems and Solutions
Name Resolution Issues
Security Problems and Solutions
Malware Troubleshooting
Too Few Permissions
Too Many Permissions
Too Much Running
Confidentiality and Integrity
Performance Optimization
Hardware Optimization
Software Optimization
Network Optimization
Hands-on Exercises
Troubleshooting Methodology
Hardware Problems and Solutions
Software Problems and Solutions
Storage Problems and Solutions
Network Problems and Solutions
Security Problems and Solutions
Performance Optimization

Working in IT means solving problems—after all, IT solutions exist to solve business problems. This guide focuses on how to troubleshoot a wide variety of issues to get people back to being productive as quickly as possible.

Troubleshooting Methodology
Solving IT issues involves a combination of knowledge and experience, plus the application of a sound troubleshooting methodology. When things that once worked no longer work, your first question should always be, “What has changed?” Finding the answer could involve reviewing server log files, consulting with other technicians, and a host of other activities.

Following is the CompTIA Server+ troubleshooting methodology:
1. Identify the problem.
2. Establish a theory of probable cause.
3. Test the theory.
4. Establish a plan of action.
5. Implement a solution or escalate.
6. Verify functionality.
7. Perform root cause analysis.
8. Document the solution.

Let’s cover these steps in more detail.

Identify the Problem
Of course, you have to be able to identify exactly what the problem is before you can resolve it. This can be difficult if it involves specific functionality in a custom app, so in some cases the end user can help by further explaining the issue.
Problems are usually reported through a help desk ticketing system, which should be customized to fit your environment. For example, if your company uses Custom App 1 and Custom App 2, these should be selectable from a drop-down list when users are creating a help desk ticket. Help desk systems can also generate reports of tickets and aggregate performance- and security-related items.
Troubleshooting over the phone requires the person reporting the issue to be clear in stating the problem. To gain clarity about the issue, technicians often ask questions of the stakeholders affected by the problem whether they are end users, other technicians, or management. Technicians ask when the problem began appearing or if there were any recent changes—not only to the server itself but to anything in the environment. For example, a change in a switch’s VLAN configuration could affect user access to a server. Technicians also offer suggestions (check your IP address, make sure you’ve clicked this and that) to help identify the issue. Finding the right solution can be tricky if user responses are unclear. If possible, technicians should remotely control the affected station.

Tip: Watch for troubleshooting questions that indirectly imply that a change was made that results in a problem. Although the change itself may not be the source of the problem, how the change was implemented may be.

Identify the Problem Scope
Problem scope is often overlooked. Who or what is affected by the issue? For example, if the problem is related to Internet connectivity, what does it affect?
- Only a single server?
- All devices on a specific network?
- All devices on all networks?

CompTIA Server+ technicians need to know where to focus their troubleshooting time.

Reproduce the Problem
To solidify your knowledge of the actual problem, it helps if you can re-create the issue to help you determine what went wrong, and where. For example, perhaps the user is required to go through a very specific series of steps on a server to request a PKI certificate, which results in an error, but the problem occurs only when the user chooses a specific certificate template. Reproducing a problem like this may involve changing system configuration settings. For this reason, it’s important that you back up all settings and data prior to making any changes. In some cases, you may need to create a server image (if you don’t already have one) if the server OS becomes inoperable. Configuration management tools such as Microsoft System Center Configuration Manager (SCCM), Chef, Puppet, and Ansible can detect settings drift and remediate them back to a standard baseline configuration.

Check the Log Files
Your company’s security policies will dictate how long log files must be retained. Log retention usually requires log rotation, which involves starting a new log file and compressing and archiving old log files.
There are many types of log files:
- Windows Event Viewer logs
- UNIX/Linux logs
- Normally stored under /var/log
- Certain apps can store logs under /var/opt
- Intrusion detection system/intrusion prevention system (IDS/IPS) logs
- Firewall logs
- VPN appliance logs
- Audit logs
- File system
- Privilege use
- Account management
- Account logon
- Client operating system logs
- OS component logs
- Group Policy
- Domain Name System (DNS)
- Dynamic Host Configuration Protocol (DHCP)

Windows Server DNS log

Log files should be stored on the device, but in a larger enterprise (especially a data center), log forwarding or a central log viewing tool is critical when logs are stored elsewhere. Server virtual machines running in the cloud can be configured to copy their logs to a central logging system, such as a Microsoft Azure cloud storage account.

The figure below shows log files stored locally on a Linux host.



Viewing log files in the Linux /var/log directory

UNIX and Linux support log forwarding through the traditional syslog dameon or its successor, syslog-ng. Filters are created to determine which type of messages are forwarded to other hosts. Windows machines can be configured with event subscriptions to forward certain types of log messages to other hosts. Log forwarding is especially important for edge devices such as VPN appliances or public web sites in a screened subnet. You should make sure there is plenty of space to store logs; otherwise, new log entries may fail to get written.
Viewing or graphing log files over time can demonstrate when a problem appears. The Windows Reliability Monitor history shown in the figure below, for example, can help.



Windows Reliability Monitor history

Read the Documentation
Your ability to solve problems quickly requires you to be very organized. If, for instance, you are responsible for maintaining Ubuntu Linux servers, you should have OS documentation readily available, perhaps through an organized list of bookmarks in your web browser.
Hardware documentation for RAID controllers, network printers, hardware firewall appliances, and the like should also be easily accessible. In some cases, if you often work with the same equipment, you should pore over this documentation so that you’re prepared—be proactive!
Implementation documentation is especially important, because it is unique for each organization. You may know everything there is to know about Microsoft Windows servers, but if you don’t know how they are configured, especially in a large network, you may as well know nothing at all! Make sure configuration documentation is secured; you don’t want malicious users getting their hands on this.
Server documentation will define which roles the server houses and how they are configured. From this information, you can also determine the server’s network configuration, firewall settings including specific settings for different zones (different parts of the network with unique firewall requirements), patch history, and other important factors.
Network documentation shows server placement and the role each server plays relative to other servers and network devices. Troubleshooting server issues without specific implementation documentation makes the process long, tedious, and less effective, and it costs much more because it requires more time to fix. It’s worth paying the up-front costs to have proper documentation.

NOTE: Experienced IT consultants will not begin troubleshooting in an unfamiliar computing environment without first taking the time to absorb implementation documentation.

Establish a Theory of Probable Cause
As server technicians, we don’t want to troubleshoot issues by first testing complex solutions; instead, we check the obvious stuff first. Is the network cable plugged in? Does the NIC link indicator light show an active connection? Is the user in a group that should have permissions to access a file? Is the user trying to access a resource from a station not joined to an Active Directory domain where resource access requires this?
This can become tricky when you rely on other parties to provide accurate information about the problem. Where possible, verify that what you are told on the phone or through a help desk ticket is, in fact, correct. For example, a user complaining about being unable to print to Network Printer B may be having problems because she’s unknowingly printing to Network Printer Z. Check the obvious!
With regard to the scope issue, once you begin to formulate a theory of probable cause, you may find that this symptom occurs on other servers or devices too.

Test the Theory
Your organization should have a sandbox testing environment used for IT configuration testing and troubleshooting.

This is often a virtual network with multiple virtual machines that mimic the production environment where possible (sometimes easier said than done). Always make sure system configuration settings and data are backed up before you change settings and test theories.
As an example, reconfiguring a host-based firewall (software firewall) on a server to resolve a network service communication issue may break other network services, or it may violate corporate security policy. Back up existing settings first, and follow proper change management procedures. Once your theory is determined to be a viable resolution to the problem, you can formulate the appropriate steps to be taken in the production environment.

Establish a Plan of Action
After you’ve identified the problem and developed of a theory of probable cause, you can test the theory. If the test shows that your theory will be effective in resolving the issue, you need to develop a plan of action. The plan includes detailed steps to take against affected systems, notification of stakeholders, and, for complex problems, an implementation timeline. Simple problem resolution is much less formal and may be entirely in the control of a single technician, who can solve the problem in seconds, such as simply adding a user to a group so that the user can access a file.

Implement a Solution or Escalate
After you have established a plan of action, it must be implemented. Depending on the scope or complexity of the plan, a team of technicians may have to be assembled and notified of their roles. Imagine a network problem created by network switch VLANs that have been incorrectly implemented in a large data center. If the problem exists on hundreds or thousands of switches, many technicians may be required to solve the problem quickly, or, where possible, an automation script may be used.
In some cases, you may need to escalate to another party with the expertise and authority required to resolve the problem. For instance, when troubleshooting Internet connectivity in a branch office, if you determine that network connectivity within your branch office is working, you may escalate the issue to technicians in headquarters, where Internet traffic is centrally routed.
TIP Remember to change one thing at a time. Changing multiple settings at once makes it difficult to determine exactly which action solved the problem. If a change doesn’t fix the issue, reverse it before moving to other tests.

Verify Functionality
Don’t be tempted to declare victory too quickly! Thorough testing is required to ensure that the problem is indeed solved.
If the problem existed within custom software, you may need to involve a user who works with the custom software daily to test functionality before you can state that the problem has been resolved. Then put preventative measures in place to ensure that the same issue doesn’t occur again. If, for example, the problem resulted from a user tinkering on his Windows workstation, perhaps locking down settings using Group Policy is an appropriate resolution.
From a consulting perspective, the problem isn’t solved until the customer is satisfied. IT service frameworks such as Information Technology Infrastructure Library (ITIL) are based on this foundation.

Perform Root Cause Analysis
When the true cause of an IT problem is known, solutions come much more quickly. At the same time, preventative measures are effective because they are focused.
Conceptually, this is similar to putting out a fire, identifying the source of the fire (such as leaving the stove’s burner on after cooking), and then preventing other fires from occurring (always turn off the burner). In the case of IT, prevention may involve preventing the use of personal USB devices to stop malware infections. In other words, you must treat the root cause and not just the symptoms.
Understand that you may be looking for multiple related root causes, not just one. To prevent similar problems in the future, remember that the key question is “Why?”

The following list offers a sense of how to deal with root cause analysis:
- Why did machines get infected with malware?
- Cause: User personal USB devices were not blocked.
- Solution: Block personal USB device use.
- Why were newly hired users unable to access required files?
- Cause: They were not added to the required groups.
- Solution: Create new users from a template that includes required groups.
- Why was file server response time brought to a crawl?
- Cause: A disk failed in a RAID 5 array so data was rebuilt on demand ( See figure below).
- Solution: Ensure that hot spare disks are always available in the RAID array so that data can be rebuilt on disk and served quickly to users.



Three 4.88GB failed disks in a Windows Server software RAID 5 array

Document the Solution
Throughout the troubleshooting process, your structured approach will eventually reveal a root cause and related remedies to prevent future occurrences. You need to document the result as well as the steps taken along the way.
Many organizations use help desk ticketing software that integrates searchable knowledge functionality. When tickets are closed because solutions were found, this information is stored with metadata in a database that can be quickly searched in the future to avoid having to go through the entire troubleshooting process again if the same problem occurs. As a result, in some cases, IT configurations change, business processes are improved, and security controls are further hardened.

Hardware Problems and Solutions
These days, when hardware is properly maintained, it is pretty resilient to problems. Proper maintenance includes applying firmware updates when they are available. There’s also the issue of overheating due to insufficient airflow or failed fans, which greatly reduces the life expectancy of equipment and can cause erratic behavior.

NOTE: Most servers and related equipment have hardware sensors that detect environmental conditions such as temperature and can predict imminent failures. You may need to install software for this data to be available.

The table below lists common hardware problems and solutions. Sometimes the problem is server add-on hardware such as expansion cards (including those for blade enclosure backplanes) that are not seated properly in their slots, or onboard components integrated with the motherboard. In other cases, some hardware components may simply be incompatible with others; refer to documentation for more information. For Windows servers, check the Hardware Compatibility List (HCL).



Common Hardware Issues and Solutions

Server+ technicians can be prepared by using a crash cart, which is a collection of tools and replaceable components used when troubleshooting servers.

You may want to keep the following hardware and software tools handy when troubleshooting server and network hardware issues:
- Multimeter to test power supplies
- Hardware diagnostic tools for memory (seen in Figure below), RAID controllers, disks, motherboards, and expansion cards
- Can of compressed air for removing dust
- Antistatic wrist straps and ESD mats
- Tool for testing bad RAM chips


Windows Server built-in memory tester


Software Problems and Solutions
More often than not, you’ll be troubleshooting software issues instead of hardware issues. Other than the OS, application software runs in an environment that could be hosting literally millions of other software components; no wonder we spend a lot of troubleshooting time here!
Again, don’t forget to check the obvious. For example, a mobile phone user may unknowingly mute a phone microphone during a conversation and assume there is a transmission or phone problem. This specific issue is called discontinuous transmission (DTX), and simple issues like this are easily avoided with user training. Many organizations expect users to learn how to use new hardware and software solutions by themselves, without having proper training, but this is not a place to cut corners!

The table below lists commons software problems, causes, and solutions.









Common Software Issues and Solutions

As a prepared CompTIA Server+ technician, you will have software utilities available at all times to resolve software issues quickly. Some restrictions, such as installing or running applications, can be related to UAC configuration, as shown below.



User Account Control settings on Windows Server

Remember to check related logs for clues. For instance, if a DNS server doesn’t seem to be responding to any DNS queries from all clients, yet it’s up and running on the network, check the DNS server and client logs.

You can monitor resource use in Windows using a variety of tools:
- Task Manager
- Resource Monitor
- Performance Monitor
- Data Collector Sets

The Windows Resource Monitor ( Figure below) gives you further insight as to which processes are consuming the most disk I/O time, often referred to as input/output operations per second (IOPS); the other tools do not provide this detail.



Viewing disk I/O statistics in Resource Monitor

Data Collector Sets (DCSs) are similar to Performance Monitor in that you can add metrics that you want to monitor. DCSs differ from Performance Monitor in that you can control when to start and stop collecting this data, and you can configure alert notifications when certain thresholds have been exceeded. For example, if CPU utilization goes beyond 80 percent for a period of time, administrators can be notified in a variety of ways ( Figure below).



An alert DCS based on CPU usage

Linux systems can be started in single user mode (run level 1), where only a minimal set of services are running. The exact process varies among different versions of Linux distributions, but it involves interrupting the boot process or modifying a boot startup file. This can sometimes enable booting and troubleshooting where otherwise the machine might not even boot.

UNIX and Linux environments provide various tools to troubleshoot software issues:
- top  Lists top processes consuming resources
- ps  Lists running processes
- kill  Terminates processes
- df  Shows disk free space

Windows environments have built-in tools for mapping drives, such as those that run within the GUI, command line tools such as net use ( Figure below), or the new-psdrive PowerShell cmdlet. You can also use the GUI to view disk, file, and folder properties by right-clicking and choosing Properties.

Mapping a drive letter in Windows with net use
Using PowerShell, the get-volume cmdlet shows file system health status as well as size statistics ( Figure 7-10).



View disk volume information using get-volume

Storage Problems and Solutions
Always remember to check the obvious—for example, loose or faulty power or data cables for disk devices can cause the drive to be unavailable or available intermittently, and we want to maximize uptime. Some disk I/O buses, such as some SCSI variants, required both ends of the bus to be terminated. Newer SCSI adapters and devices are self-terminated, whereas older equipment requires a physical terminating device.

Storage media sometimes has storage areas that do not reliably store data; disk-
scanning tools will mark these areas as “bad sectors” or “sector block errors” to prevent the server OS from attempting future disk writes to these locations. If the entire disk partition (or volume, if it spans multiple disks) is corrupt, or if users do not have the correct permissions, they may receive a “cannot access logical drive” message. The same is true for servers attempting to access network storage.
Occasionally, software issues will appear to be disk-subsystem issues. Consider, for example, what happens when a power failure occurs in the midst of applying server OS patches that modify the boot sector. Chances are the boot sector will be corrupt and the server will no longer boot, which, in Linux, can result in a “kernel panic.” Luckily, modern server OS installation media often provides a repair option (Windows) or a boot Rescue Mode (Red Hat Enterprise Linux) to deal with such situations.

A failing CMOS battery can cause the server to forget its hardware configuration, including the date and time. This eventual time skew can prevent network services from talking to this server, and this can also be the source of disk boot configuration errors. Some clues regarding drive failures are obvious—hearing a light metal scraping sound as a hard disk fails or noticing the odor of electrical components burning when a circuit board is fried.

This table lists common storage problems, probable causes, and possible solutions.



Common Storage Issues and Solutions

Windows Tools
You can manage and troubleshoot disks on Windows servers in a number of ways. In some cases, you may need to wipe and repartition a problematic disk—just make sure you’ve made a backup of the disk contents beforehand.
 

Command line tools include the following:
- Diskpart.exe
- Defrag.exe
- PowerShell cmdlets

Note: Newer Windows OSs (such as Windows 10 and Windows Server 2019) no longer include the old fdisk command line utility; it has been replaced with diskpart.exe.

At the GUI level, you can use tools such as these:
- Disk Management
- Server Manager
- Disk Defragmenter
- Disk Cleanup
- Error Checking

The following set of diskpart commands wipe (clean) the third disk in a machine and then convert the disk to a GUID Partition Table (GPT), which enables larger capacity partitions as well as a larger number of partitions than a master boot record (MBR). (See this guide) A 50GB primary partition is created, formatted as NTFS, and assigned drive letter E:\.

Using Microsoft PowerShell, you could initialize, partition, and format a disk, as shown in the following example:


 

PowerShell also includes cmdlets for troubleshooting file system issues:
Repair-Volume -DriveLetter E –OfflineScanAndFix

Linux Tools
Like Windows, some UNIX and Linux variants provide a bootable rescue mode to troubleshoot disk boot problems, often from OS installation media. From rescue mode, or in some cases, within the OS itself, you can use tools to manage and troubleshoot at the command line, including the following:
- df  Shows disk free space
- fsck  Checks file systems for corruption
- xfs_repair  Checks for and repairs an XFS file system
- iostat  Shows disk I/O statistics for storage devices
- lsof  Lists open files and provides further details
- mdadm  Linux software RAID array management

For example, to repair an unmounted XFS file system on /dev/sdc1, as shown below, you would use this command:



Repairing an XFS file system using xfs_repair in Linux
xfs_repair /dev/sdc1

Sometimes a lack of free disk space can cause many, often performance-related, problems. Monitoring disk capacity is crucial.

The Linux disk free (df) command ( Figure below) has a -h (human readable) switch to show file systems and their disk space usage statistics in terms of megabytes, gigabytes, and so on, instead of in units of 1KB blocks:



Viewing disk free space in Linux using the df command
df -h

To determine which Linux disk device has the most I/O activity, use the iostat command, as shown below.



Viewing disk I/O statistics using the iostat Linux command

Network Problems and Solutions
Network problems can be hardware related, software related, or both. For example, physical network cables can sometimes have problems. A time-domain reflectometer (TDR) can be used to measure the continuity of electronic signals through circuit boards and the wires within a network cable to identify faults.

Tip: Don’t confuse TDRs with tone and probe generators on the exam or on the job. TDRs are used to identify where cable faults exist. Tone and probe generators are used to identify specific cables in large cable bunches by sending a tone through wires in a wall jack and identifying that signal at the other end of the cable in a wiring closet.
Optical time-domain reflectometers (OTDRs) result in traces that are used to show where fiber-optic cables are terminated and can show locations of cable breaks. OTDRs are expensive devices that require expertise to use and to interpret results.
Many network problems stem from incorrect software protocol configuration. IPv4 and IPv6 addresses must fall on the correct subnet to function properly. The default gateway, or router, is used to send traffic outside of the LAN to other local and remote subnets; it must be reachable on the LAN.

Tip: You may see at least one network architecture diagram-based question testing your knowledge of valid IP addresses, subnet masks, and which router interface your default gateway configuration should point to.

This table outlines the reasons for common network issues and potential solutions.



Common Network Issues and Solutions

When destination hosts on different networks are unreachable, the IP address, subnet mask, and default gateway must be checked to ensure that their values are correct. Incorrect routing information will result in “destination host unreachable” messages. If IP settings are correct, the Windows tracert or Linux traceroute command can be used to determine how far down the line (through routers) transmissions are getting before encountering problems. This is more useful than the ping command, which would simply report that the host is unreachable. (Note that Windows machines use the ping -6 command to use IPv6, such as when pinging a hostname; Linux systems uses the ping6 command.)
EXAM TIP The tracert command uses ICMP as its transport mechanism. This command will not be a useful troubleshooting tool if host or network firewalls block ICMP traffic. Carefully read exam questions that mention this issue to avoid choosing the incorrect answer.
IPv6 troubleshooting is not that different from troubleshooting IP4. Similar to how the IPv4 subnet mask is compared to the IPv4 address, devices with the same IPv6 subnet prefix are considered to be on the same network. On a Windows server, ipconfig shows both IPv4 and IPv6 information.
The Windows route command (ip route show in Linux) can be used to display or modify routing table entries on a Windows server. For example, to display routes, you would use route print, as shown below. Take note that the 0.0.0.0 route is the default route; notice the IP address of the default gateway on the same line.



Output from the Windows route print command

To ensure that network services are listening on a specific TCP or UDP port, use the Windows netstat command. For example, if an HTTP web server is not running and therefore not listening on TCP port 80, clients will be unable to connect to the web server.

Name Resolution Issues
If names are resolving to unexpected IP addresses, it means there are probably entries in the local HOSTS files on the system. The HOSTS file entries are placed into the client DNS cache in memory, and this cache is checked before DNS servers for name resolution.
Recent answers to DNS queries are also cached in the local client DNS memory cache and have a time-to-live (TTL) value that determines how long the entry remains cached. The Windows ipconfig command includes a /flushdns parameter that clears out the client DNS cache; use this when the DNS records have changed recently.

For further DNS troubleshooting, including solving “unknown host” name resolution errors, use the nslookup command. After typing nslookup and pressing ENTER, you will be in interactive mode. You can use the server directive to connect to a different DNS server. The set type= command changes the type of DNS resource record you want to test, as shown below.



Using nslookup to view mail server information through MX records

Security Problems and Solutions
Plenty of problems arise from security compromises, from loss of critical business data, to reputation loss and potential lawsuits, and the list goes on and on. Many (not all) security problems are preventable, however, and this section will discuss how to troubleshoot security problems and how to prevent them from recurring.

Malware Troubleshooting
In addition to malware notifications, symptoms of malware infection include the following:
- Excessive and prolonged hardware resource use
- Inability to reach network resources
- Web browser home page is changed and is not editable
- Web browser opens pages user didn’t navigate to
- Rogue processes or services (mystery processes) running with improper privilege escalation
- Missing log entries (cleared by the attacker)
- Encrypted files with a message demanding payment
- Abnormal listening ports on the server, which could indicate a backdoor created with tools such as netcat (nc)

User awareness and training is the number one defense against most security threats. This guide discussed social engineering and user awareness regarding network malware attacks.
When infections are detected, the server or subnet must be isolated immediately; this can be done manually or automated through your antimalware solution. You can specify a script that runs on detection. The script may look at the subnet address and disable routing to and from the subnet, or it may disable the switch port that links that subnet to the rest of the network. Many malware infections sometimes attempt to connect to a command and control (C&C) server outside of the local network, in some cases using HTTP or DNS packets in an attempt to make the traffic blend in with normal traffic. Configure your network IDS and IPS solutions accordingly.

NOTE: Servers should never be used as desktop computers. Tasks such as checking e-mail, reading the news, or downloading drivers should never be allowed directly on a server.

Malware Removal
The removal of malware can be done in many ways:
- Vendor malware removal tool
- Windows system restore point
- Client OS only, not Windows Server OS
- Revert to previous configuration, not data
- Server reinstall or reimage
- Ensure that you have a working backup
- Ensure that the image being restored matches the available physical or virtual server hardware configuration
- As a last resort, manually restore the OS
- Boot through alternative means to remove the infection
- Windows safe mode
- USB boot to run removal tool
- PXE boot to run removal tool

You should use more than one antimalware product to ensure that remnants of the infection do not remain; however, do not run multiple antimalware products at the same time, because some infections could be missed because of conflicts. Keep in mind that even if multiple scans do not detect an infection, this doesn’t mean the system is not infected; a zero-day attack may be the culprit, for example, which may not be detected by a malware scanner.
Antimalware heuristic analysis is useful in detecting previously unknown malware by analyzing suspicious behavior, similarly to how IDS and IPS systems watch for abnormal host activities or network traffic. The downside is false positives, the reporting of problems when they do not exist.

Too Few Permissions
This guide discussed file system permissions for Windows and Linux systems. Normally, users are added to groups, and groups are added to ACLs and either granted or denied permissions.

Permissions in Windows
On Windows servers, combining share and NTFS permissions results in the most restrictive permissions being applied, and this is a big part of troubleshooting file system issues in Windows environments. In the GUI, you can view the properties of a file or folder to see the effective permissions for a given user or group if a user is unable to open a file.
Too few permissions can also mean applications won’t launch. Windows systems may need UAC settings loosened up to enable apps to run. Some apps require administrative privileges not only to install but also to run—it depends on the app and where it reads and writes to and from in the file system and registry. On a Windows server, you can hold the SHIFT key while right-clicking a program file such as an .exe file to see the “Run as administrator” or “Run as different user” options.
For Windows machines, Local Group Policy executes before Active Directory Site, Domain, and OU Group Policy settings. Group Policy is often used to harden Windows clients and servers, but sometimes it may be a little too restrictive, and it’s important that you know how to view effective Group Policy Objects (GPOs) for a server to troubleshoot this type of issue. While the gpudate command forces a Group Policy update, gpresult /r shows the resultant set of policy, listing GPOs that are applied to the machine where the command was run.

The figure below shows the output from gpresult /r.



Output from the Windows gpresult /r command

Security filtering enables Group Policy administrators to ensure that only specific users or computers get Group Policy settings. This can be done by specifying individual users, computers, or groups, or by using Windows Management Instrumentation (WMI) filters. WMI filters use the WMI Query Language (WQL), as shown in the following example, where Microsoft Hyper-V and VMware virtual machines are being selected:

Tip: Exam questions may imply that central configuration changes (Group Policy) have been made and that they have affected some or all users or computers. Remember that “some” computers or users could be very specific, as shown in the WQL example. WQL can query any hardware or software attribute related to users or computers.

Privileges in Linux
Linux administrators can use the sudo command prefix in front of a command that requires elevated privilege—of course, sudo has to be configured in the first place to enable this. The following command enables a sudo-authorized regular user to set the password for bjones using sudo:
sudo passwd bjones

Another consideration in Linux is the SetUID special bit, which enables an executed script or binary to run not as the invoker, but as the file owner (which could be root). This warrants very careful attention, but it can solve the problem of a regular user not being able to run a script or program.

The following example adds the SetUID bit to a file called /backupscript.sh. If someone can execute the script, it will run with the permissions of whoever the file owner is (find this out by using ls -l).
chmod u+s /file.txt

Restoring File System ACLs
You can save file system ACLs and restore them if you run into trouble later. The icacls command works well for this on Windows servers; getfacl and setfacl work well in Linux.
Windows ACLs  In this example, the first command displays ACL entries for user DLachance in and under E:\Projects.

The second command saves the ACL entries to a file, and the third command restores ACLs to the file system from the Project_ACLs_Server1 file.

If users were once able to open files and now cannot, ACLs may have been changed, so you can use icacls in this case.

Linux ACLs: The Linux getfacl and setfacl commands work well for saving and restoring ACLs. In the following examples, the first command recursively gets file system ACLs from /budgets and saves to a file called /budget_acls; the second command restores ACLs from the saved file.




Too Many Permissions
When someone needs permissions quickly to do something in Windows, it’s tempting to add the user to the Administrators (local) or Domain Admins (Active Directory) group to expedite the request, but this is blatant disregard for the principle of least privilege.
Users must be granted only those permissions required to complete a job task and nothing more. Of course, this prerequisite may take a bit of research and testing on your part, but it’s worth the up-front time investment to grant permissions correctly. A security audit can reveal this problem, or you can proactively check into it, perhaps by using commands such as icacls or getfacl, or by using a third-party security auditing tool.

Too Much Running
Modern server OSs are bare-bones, meaning they don’t have many extra software components installed by default, so you have to install them. This is a good thing, because having too many services running simultaneously on a single server can negatively affect performance.
Port-scanning tools (such as the one shown in Figure below) should be used periodically as a proactive measure to determine which ports are open on hosts. This gives you a picture of which network services are running, which is related to which firewall ports should be open on both host-based and network-based firewalls.



Results of a port scan against a Linux host

Of course, having many services running also increases the attack surface and increases the amount of time required to patch the server—so keep it minimal when possible!

Confidentiality and Integrity
Encryption provides confidentiality. A cipher is a cryptographic algorithm that is used to scramble, or encrypt, and decrypt data. Use of an incorrect cipher can cause problems. Some hardware appliances and software configurations require the use of certain ciphers. For example, server file encryption software may be able to decrypt AES 128-bit files but not AES 256-bit messages, so cipher strength can be a problem.
The same type of situation could occur at the network level. You may be using an older web browser that does not support the encryption required by the web server. Yet another web browser problem may be PKI certificate trust: When connecting to an HTTP web server, your browser checks the signer of the server’s PKI certificate. If the web browser does not trust the signer, it does not trust the PKI certificate used by the web site. This is solved by adding a new trusted certificate signer to the web browser device.
Integrity uses a hashing algorithm such as SHA-256 to ensure the trustworthiness of data, in that it is authentic and has not been tampered with. Authentication can occur at the network packet level or at the file level. You can use packet sniffing (capturing) tools such as Wireshark to analyze packets on the network as well as bad packet checksums. Packet checksums are used to ensure that what is received is what was sent, and, if not, a retransmission may occur (which depends on the protocol being used).

Packet sniffing can also reveal the use of insecure tools such as Telnet ( Figure below), which transmits data in clear text.

The use of insecure tools can introduce malware or could serve as an attack vector for malicious uses—more trouble that we don’t need!



Viewing Telnet captured network traffic in Wireshark

Files can be hashed using various tools. A file hash is a unique value that represents the state of a file.

If anything about the file changes, when we generate a new hash, it won’t match the old one; this tells us that something has changed.
Microsoft PowerShell supports file hashing using the get-filehash cmdlet:
get-filehash .\Project1.txt

The Linux sha256sum command, shown in the figure below, can be used to generate and verify file hashes. Notice that the first hash of samplefile1.txt (beginning with 0356) differs from the second hash (beginning with f9b5), because a change was made to samplefile1.txt (the text “More sample data” was appended to samplefile1.txt).



Hashing files in Linux using sha256sum

Performance Optimization
Performance optimization applies to hardware, software, and specific implementation settings. It’s important that you always monitor your environment for any changes in performance so that you can take action before you end up troubleshooting major issues. Plenty of enterprise-class network and server monitoring tools are available, such as Spiceworks and Microsoft System Center Operations Manager, to name a few.
The Windows Performance Monitor tool ( Figure below) is built into the Windows OS. It enables you to add items to monitor, such as network components, specific software components, hardware items such as CPUs and physical memory, and more. You can also use Performance Monitor to reach out over the network to monitor servers.



Windows Performance Monitor

Linux administrators commonly use tools such as ps, top, and iostat for performance monitoring.

Hardware Optimization
Naturally, getting the fastest and most reliable hardware is always desired. More CPU cores and more RAM (especially for hypervisor servers) all make a difference in performance.
Since disk I/O often tends to be the performance bottleneck, it warrants our attention when it comes to optimizing performance. As discussed in this guide, solid-state drives (SSDs) perform better than magnetic hard disk drives (HDDs), and you can configure them both in storage tiers so that frequently accessed data is stored on the faster SSDs.
Whether server storage is local or accessed via a storage area network (SAN), hardware RAID levels designed for performance (disk striping, or RAID 0, for example) should be used. In the cloud, selecting the number of virtual CPUs, amount of RAM, and disk type for a virtual server is referred to as sizing.

Software Optimization
Having fast hardware is important, but so is the software configuration that uses it. For example, adding more RAM to a hypervisor server makes sense, because it will host multiple virtual machines, but at the software level, each virtual machine may be configured with dynamic memory. This means virtual machines currently needing RAM can “borrow” it from those virtual machines that do not currently need it. Allocating too much RAM statically to one virtual machine can starve others from memory.
Less is more. Keep your server OS clean, and install only the components that are needed. Not only will this keep your server running quickly, but it reduces the attack surface, decreases patching time, and reduces the overall amount of troubleshooting. The more stuff you have installed and running, the greater the likelihood of something going wrong.

Network Optimization
In addition to getting faster network equipment (switches, routers, NICs) or Internet connections, you can improve network performance with what you already have.
VLANs can be configured within network switches to group machines that communicate often with one another into smaller networks. Breaking large networks into smaller ones makes network transmissions much more efficient and essentially speeds up the network.
Servers normally have multiple NICs. You might consider configuring NIC teaming to group NICs together for better network throughput into and out of the server. Bear in mind that network switch ports where the server NICs are plugged in must be configured so that inbound server traffic can take advantage of NIC teaming.
Network load balancing (NLB) distributes incoming traffic for network services such as a web site to multiple back-end servers running the same service and holding the same content. For example, you might have four back-end web servers configured the same that replicate web site content to one another. The public IP address of the NLB is what DNS FQDNs would resolve to, not the back-end server IPs. Load balancers can improve the performance of an on-premises application or a cloud application.
One great thing about this in the public cloud is elastically scaling the number of back-end servers as needed behind the NLB, such as in response to increased demand. This is called horizontal scaling. The end result is that the network service responds quickly.

Notice in the figure below that load balancers listen on a specific public port—in this case, TCP port 80 for an HTTP application—and that the configuration specifies an internal port number that could be different.

Also notice in the figure that step 5 of the wizard enables you to add multiple EC2 virtual machine instances (the back-end servers).



Creating an elastic load balancer in the Amazon Web Services cloud

You should periodically capture network traffic and examine the capture results to ensure that only required network protocols are in use.
 

Hands-on Exercises

 

Exercise 7-1: Explore Windows Performance Monitoring Tools
1. Ensure that you are logged into Srv2019-1 with the Domain Administrator account (Fakedomain\Administrator) with a password of Pa$$w0rd172hfX.
2. From the Start menu, enter perf and then click Performance Monitor.
3. On the left, expand Monitoring Tools if it’s not already expanded, and then click Performance Monitor. The %Processor Time counter is automatically shown on the graph to the right.
4. Click the green plus sign at the top of the screen to add monitored items.
5. Scroll within the alphabetized list, expand Physical Disk, and choose Avg. Disk Queue Length. At the bottom left, choose C: (it may be listed with a number, such as 2 C:) and click Add. Then click OK. The Avg. Disk Queue Length counter is added to the chart (view the counter underneath the chart). Disk commands that cannot be serviced immediately due to a busy disk are queued up, which is normal in most environments. The specific value that should be of concern here will vary in different environments.
6. Start Windows Explorer. Navigate to C:\. Right-click C:\Program Files, choose Copy, and then press CTRL-V to paste. This will generate disk I/O activity.
7. Switch back to Performance Monitor and notice the graph spikes. Hover your mouse pointer over the highest graph spike to display the specific metric that the line represents.
8. Using Windows Explorer, delete the copy of Program Files on C:\.
9. Switch back to Performance Monitor. On the left, expand Data Collector Sets. Right-click User Defined and choose New | Data Collector Set. Enter Notify of CPU spikes for the name.
10. Choose Create Manually (Advanced).
11. Click Next and select Performance Counter Alert. Click Next.
12. Click Add. Scroll down to the Processor category and expand it. Choose %Processor Time, click Add, and then click OK.
13. At the bottom, specify Above 90 for the Limit value and click Next.
14. Choose Open Properties For This Data Collector Set and click Finish. In the Notify Of CPU Spikes Properties dialog box, click the Schedule tab, click Add, and set the beginning date to today’s date. Click OK twice.
15. Double-click Notify Of CPU Spikes in the list. Double-click DataCollector01. Set the sampling interval to every 15 minutes.
16. Open the Alert Action tab and check the Log An Entry in the Application Event Log option. Then click OK.

Exercise 7-2: Explore Linux Performance Monitoring Tools
1. Ensure that you are logged into Ubuntu-1 as uone with a password of Pa$$w0rd172hfX. Press ENTER after each command you type in the following steps.
2. Enter df -h and notice free disk space on disk devices under the Avail column as well as the Use% column.
3. Enter ps -aux | grep sshd to view SSH daemon process information only. The grep command is a line-filtering command. Notice the PID (column 2) and the %CPU utilization (column 3). If you see “Zombie” processes, these are processes that have finished executing but are waiting for the parent process to complete.
4. Enter top. Notice that the display changes every few seconds.
5. Press S and then enter .5 to increase the update frequency to every half-second. Press ENTER.
6. Press Q to exit.

Exercise 7-3: Use Microsoft PowerShell to Retrieve Event Log Information
1. Switch to Srv2019-1.
2. Start PowerShell.
3. Enter get-eventlog system. Notice the long list of event log entries. You can interrupt this if it takes too long by pressing CTRL-C.
4. Enter get-eventlog system -newest 5.
5. Enter get-eventlog system -entrytype warning | more. Warning log entries are displayed. Press CTRL-C.
6. Enter get-eventlog security -newest 10 -computername Srv2019-1. Remote security log entries are displayed from Srv2019-1.
NOTE The -computername parameter is used here for demonstration purposes only. It is not needed because we are already at Srv2019-1 issuing the command. You can specify multiple computer names in a comma-separated list.

Exercise 7-4: View Linux Log File Entries
1. Switch to Ubuntu-1.
2. Enter dmesg | grep scsi to view SCSI disk device startup messages.
3. Enter cat /var/log/syslog | more to review the Linux system log file one screen at a time. Press the SPACEBAR to advance one line at a time. Press Q to quit.
NOTE To view real-time log file updates in Linux, use tail –f. For example, tail -f /var/log/syslog. Press CTRL-C to exit the viewing.

Quick Review:
Troubleshooting and performance optimization are linked. Poor performance indicates that you should troubleshoot the case and remedy the issue. Follow a structured approach to solve problems as quickly and efficiently as possible.

Troubleshooting Methodology
Follow the standard troubleshooting steps:
1. Identify the problem.
- Determine the scope.
- Question stakeholders.
- Reproduce the problem.
- Consult log files and documentation.
- Check the obvious.
- Use a test environment.
- Notify stakeholders.
- Establish an implementation timeline.
- Change one thing at a time.
- Revert ineffective changes.
6. Verify functionality.
- Test thoroughly.
- Satisfy the end user.
7. Perform a root cause analysis.
- Determine the source of symptoms.
8. Document the solution.
- Keep a record during all troubleshooting phases.

Hardware Problems and Solutions
Hardware problems are quickly fixed with proper documentation (OEM and implementation), as well as tools such as multimeters to test power and diagnostic tools to test memory and RAID controller health and configurations. Other hardware issues to watch out for include the following:
- Apply firmware patches.
- Ensure that firmware in use is compatible with other hardware components.
- Ensure that proper HVAC is in place.
- Determine any hardware incompatibilities.
- Look for component failures (sometimes identified during POST).
- Look for boot failures.
- Check power issues.

Software Problems and Solutions
Software troubleshooting includes the operating system, drivers, and applications. Resolving software issues includes taking actions such as these:
- Apply software patches.
- Operating system
- Drivers
- Applications
- Reset the user password.
- Delete corrupt user profiles.
- Ensure that user permissions are set correctly for user and service accounts; restart services if making configuration changes.
- Run antimalware scans.
- Use monitoring tools to identify memory leaks or runaway processes.
- Boot into safe mode, repair mode, or rescue mode.
- Ensure that dependent services are running.
- Restart problematic services.

Storage Problems and Solutions
Physical storage device and cable issues can sometimes be the source of problems. Hot spare disks in disk arrays provide a quick way to resolve failed disk issues. Other items include the following:
- Disk I/O bus termination
- Failed disk adapters
- Failed disks
- Failed network path to network storage
- Insufficient disk space
- Insufficient user permissions to access storage
- File system corruption
- Slow file access
- Incorrect use of RAID level

Network Problems and Solutions
Physical network components such as NICs, switches, routers, and wireless access points can fail. Firmware and driver updates should be applied for optimal stability and functionality. TDRs and OTDRs can be used to ensure copper-based and fiber-optic cable continuity. Other network issues include the following:
- Incorrect IP address, subnet mask
- Incorrect default gateway, DNS server
- DHCP/DNS server unreachable
- Firewalls (network ACLs) blocking legitimate traffic
- Incorrect VLAN membership
- Mismatched NIC speed settings

Security Problems and Solutions
Documentation regarding the implementation of security controls reduces time spent troubleshooting. Other security issues include the following:
- Malware infections
- Inadequate authentication as required by server
- Untrusted or expired PKI certificate
- Insufficient permissions to access a network resource
- Too many permissions granted beyond what is required
- Incorrectly assigned share, NTFS, or Linux file system permissions
- Group Policy settings enforcing too little or too much security
- Linux users not using the sudo command prefix to run elevated commands
- Unnecessary installed software
- Unneeded running services
- Incorrectly configured network firewall ACLs
- Mismatched cryptography ciphers
- Inability to decrypt transmissions or files

Performance Optimization
Performance optimization can result from replaced components, changed configurations, and ongoing monitoring of resource use.
Windows monitoring tools include the following:
- Reliability history
- Log files

Linux monitoring tools include these:
- ps
- top
- iostat
- df

Network optimizations include NIC teaming, VLAN traffic isolation, and network load balancing.