CompTIA Security SY0-601 Exam: The Basics of Incident Investigation

Objective: Given an incident, utilize appropriate data sources to support an investigation.

Topics:
- SIEM dashboard
- log file
- syslog
- journalctl
- nxlog
- bandwidth monitoring
- metadata
- NetFlow/sFlow

Suspected incidents or indicators require analysis and validation. When the response team has determined that an incident has occurred, the next step is to take a comprehensive look at the incident activity to determine its scope. A proper determination of the scope of the incident helps the team prioritize potential needs for deeper analysis, as well as the next step in the process for containment. To help with prioritization efforts, the response team should consider and categorize the impact and recoverability effort. A number of tools are available to support these investigations.

SIEM Dashboards
“Security Assessment Techniques” provided an overview of security information and event management (SIEM) solutions. Recall that SIEM solutions provide a number of useful purposes, one of which is supporting incident response. The primary function of a SIEM system is to aggregate data from various sources. Such a system can therefore be used to aggregate data on an incident, based on source IP, destination IP, and event ID. Having the raw data aggregated together in a single spot is valuable for incident investigation. Furthermore, the correlation engine part of a SIEM system can help uncover important information relevant to the investigation. It is important to consider the events of interest and other factors that led to the requirement for an investigation. Sensitivity is an important factor to consider as a SIEM system combines indicators with other techniques, such as pattern matching and anomaly detection. False positives waste investigative efforts.
A SIEM system supports an investigator by providing intelligent correlation. Unlike simple logging, a SIEM system can tie together individual benign events to reveal more details about what occurred and why it’s of concern. For example, a proxy log might show that a user successfully authenticated and is currently making outbound web requests. At the same time, a physical security access log might show that the same user left the office some time ago. By themselves there isn’t anything too concerning about either of these events. When taken together, however, they indicate an issue. As another example, say that you enter a password incorrectly the first time you try to log in to an application. That application logs the invalid login attempt, and the SIEM system sees it, too; by itself, the failed login is not a problem. If your user account also incorrectly tried to log in to 20 other applications around the same time, however, that is a problem, and the SIEM system can see it. These correlations a SIEM system can make become critical to creating meaningful alerts and helping incident investigators.
Individual logs are useful for providing indicators of incidents and supporting security incident investigations. A SIEM system is able to correlate across these sources to help an investigator validate and further research an incident.
Dashboards and trending also provide meaningful and easy-to-visualize information based on correlated data. SIEM systems provide comprehensive charting capabilities to identify patterns and abnormal activities. Modern SIEM solutions also go beyond just static visualizations. A static image can certainly help identify trends and potential incidents, but it’s the ability to drill down that really helps an incident investigator. SIEM systems provide interactive dashboards with charts, sensors, graphs, and trends that can be easily explored by automatically filtering and exploring the data through interactive drilldowns. Figure 28.1 shows an example of a SIEM dashboard.



An example of a SIEM system security dashboard

Logging
Logging is the process of collecting data to be used for monitoring and auditing purposes. Log files are documentation, but how do you properly set up a log? You should develop standards for each platform, application, and server type to create a checklist or monitoring function. When choosing what to log, carefully consider your options. Logs take up disk space and use system resources. They also have to be read; if you log too much, the system bogs down, and weeding through the log files to determine what is important takes a long time. Be sure to mandate a common storage location for all logs. Documentation also should state the proper methods for archiving and reviewing logs.

All devices, operating systems, and applications have log files that generate important data, such as authentication attempts and errors. For example, application log files contain error messages, operational data, and usage information that can help manage applications and servers. Analyzing web application logs enables you to understand who visited the application, on what pages, and how often, and it also provides information on errors and performance problems of the web application. Analysis of logs from web servers such as Apache, IIS, ISA, and Tomcat is automatic and can contribute important insight into website and web application quality and availability. Web server logs are usually access logs, common error logs, custom logs, and W3C logs. W3C logs are used mainly by web servers to log web-related events, including web logs. Operating systems can also generate dump files. Such a file provides a log of the different processes running and being loaded for the system at a particular point in time. These files may also include data related to what was loaded into memory.
Unlike a security log, the application and system logs are available for all users to view. You can use the application log to determine how well an application is running. The system log shows events that occur on an individual system. You can configure settings such as the size of the log file and filtering of events. Event logging is used for troubleshooting or notifying administrators of unusual circumstances. Be sure that you have the log file size set properly, that the size is monitored, and that the logs are periodically archived and cleared.
Carefully consider where you store log files to keep intruders from gaining access to them. By doing so, you eliminate the ability for intruders to cover their tracks.

The table below lists and describes the fields and their descriptions for Windows events from the Windows Event Viewer.

Field Names and Associated Description for Windows Events

The Event ID and Description fields are especially important. The Event ID field makes it easy to research an event in the Microsoft Knowledge Base, and the Description text usually explains what happened in simple language.
Other operating systems also have built-in and downloadable tools that enable you to view statistics about the system, such as CPU usage, memory usage, hard drive space, bandwidth usage, temperature, fan speeds, battery usage, uptime, and the top five processes. In addition, third-party programs can monitor network health. These programs can monitor the entire network, including devices such as modems, printers, routers, switches, and hubs.
To monitor the health of all systems, you install agents on the machines and then monitor those agents from a central location. For example, Simple Network Management Protocol (SNMP) is an application layer protocol whose purpose is to collect statistics from TCP/IP devices.

The nuances of system logging and monitoring are varied and detailed, but system logs are broadly classified as follows:
- System event logs: These logs record the events that occur across the system and, most notably, that are related to the operating system. Keep in mind that these logs are specific to the system, not the user interacting with the system. Examples include hardware failures, drivers that do not load properly, and issues related to performance.
- Audit logs: Audit logs help ensure proper process and provide a useful record for auditors. Such logs provide security information such as data on successful and unsuccessful login attempts, user creation and deletion, log data deletion, user privilege modification, and file access. These logs also provide accountability and, in the case of an incident, give a record of what occurred for forensics and recovery purposes.
- Security logs: These logs contain events specific to systems and application security. Security solutions deployed within the network—for example, anti-malware software, intrusion detection systems, remote access software, vulnerability management software, authentication servers, network quarantine systems, routers, and firewalls—are a major source of such logs.
- Access logs: These logs provide information about requests and connections between systems. This can include, for example, connections between an LDAP client and a directory server (which might include details such as the IP address) and records related to the binding operation. Web servers are another common source of access logs. For example, a web server logs access to each resource, such as a page or an image. Included in the log entry are details such as IP address, browser, operating system, referring page, and a date and time stamp.

System logs vary. Be able to recognize event, audit, security, and access logs and be sure you understand when each would be used in specific scenarios.
The figure belowshows an example of Windows Event Viewer, from which Windows logs can be viewed; this figure shows the Security log. Systems generate sizable logs. When dealing with a large volume of logs and events, anomalies occur, and it can be difficult to find or recognize the anomalies. Intrusion detection system logs tend to have better automatic processing to find anomalies in network traffic than system and event logs.



Microsoft Windows Event Viewer Security log

To detect log file anomalies, the data needs to be collected, cleaned, structured, and then analyzed. Anomaly detection deduces dynamic thresholds by learning a baseline or pattern of events. Data mining techniques are often used in the analysis phase. Because systems don’t always stay static, dynamic rules can be combined with data mining techniques to produce valid anomalies.
The capability to correlate information across various data sources is paramount. Event correlation helps identify anomalies and behaviors that warrant additional investigation. Although anomaly-detection tools can automate anomaly detection in logs, it is important to understand the system. Sometimes manual analysis is required to detect anomalies.

The following are a few other tools related to logging that you should be familiar with:
- Syslog: Syslog is a decades-old standard for message logging. It is available on most network devices (such as routers, switches, and firewalls) as well as printers and Unix/Linux-based systems. Over a network, a syslog server listens for and then logs data messages coming from the syslog client, which can help in identification and detailed investigation of security incidents. Rsyslog and syslog-ng are similar but build on the capabilities of syslog by adding support for advanced filtering, configuration, and output.
- journalctl: Journalctl is a tool for querying journald, which is the logging service for Linux-based systems. Unlike syslog, journald can’t log to remote locations, but it provides a mechanism to easily forward logs to syslog.
- nxlog: nxlog is used for centralized logging across various platforms, supporting myriad different log types and formats.

The Security+ exam might ask a question regarding a log you aren’t familiar with. Take your time to examine the sample log. Particularly note the source and destination IP addresses, which may be abbreviated SRC and DST. The terms local and remote might also be used. In addition, a log might show an IP address as a source pointing to the destination with the notation -> between the source and destination.

Network Activity
An incident investigator can take advantage of a number of tools related to monitoring network activity, which is usually required as part of incident analysis and investigation. Three common types of tools are bandwidth monitors, network flow data, and protocol analyzers.
A protocol analyzer is a good tactical tool for an investigator, but it lacks the ability to scale across an entire organization, given the need to target the implementation of probes in specific areas of the organization. For an investigation, however, this may not be an issue. Network flow activity, on the other hand, can complement direct packet capture.
Network activity monitors essentially provide metadata about network communications. Metadata such as source and destination and protocol information are vital for investigations. Metadata may not in and of itself seem particularly valuable or sensitive, but it can provide valuable clues that help in piecing together details and activity. Metadata is simply data that provides information about other data. Metadata is very useful in regard to investigations, and practically everything digital contains metadata, including files and email. A digital photograph, for example, contains metadata on file type, size, and dates. In addition, a photograph taken with a mobile device may also contain the location where the photo was taken, what type of device was used, and all the camera settings.
Bandwidth monitoring tools are useful for identifying DDoS attacks, malicious host activity, data exfiltration, and other suspicious behavior. They help identify interfaces, applications, users, and protocols that are consuming bandwidth. Bandwidth monitoring tools can be as simple as those available as part of every operating system to more enterprise-oriented networking solutions, such as those that are part of a network flow solution. Figure 28.3 shows the Activity Monitor, which provides bandwidth monitoring for macOS. This utility can be useful, for example, for quickly understanding what applications or services are having a large impact on the network bandwidth.

In the figure below, you can see that the traffic is sorted by bytes received. This is not necessarily a security issue, but it indicates that someone is streaming music!



Activity Monitor for macOS

Protocol Analyzers
Protocol analyzers, also known as packet sniffers, help you troubleshoot network issues by directly capturing packet-level information across the network. These applications capture packets and can conduct protocol decoding by turning the information into readable data for analysis. Protocol analyzers can do more than just look at packets. They are useful in many other areas of network management, such as monitoring the network for unexpected, unwanted, and unnecessary traffic. Protocol analyzers can identify individual protocols, specific endpoints, and sequential access attempts. These applications capture packets and can conduct protocol decoding, which involves turning the information into readable data for analysis. Protocol analyzers are often used for network management and monitoring, and they are also valuable tools for security investigators. A network running slowly could indicate an attack, and a protocol analyzer can be used to identify port information and traffic type, which may be useful in an investigation to understand malicious activity.

Network Flow
NetFlow was originally introduced by Cisco. The term has since become synonymous with any network flow monitoring utilities. NetFlow collects packets as they are entering or exiting a network, via a router interface. A flow is considered to be a set of packets in a specific period that share common characteristics, such as the same source, destination, and protocol. NetFlow collects the packets, which are then provided to a NetFlow collector. A NetFlow application can then analyze the data, which is useful for incident investigations for intrusion analysis. An alternative to NetFlow is sFlow (sampled flow), which is also used for network monitoring. sFlow samples packets and provides analysis to identify unauthorized network activity and investigate DDoS attacks. Network flow activity builds a summary of the communication between two points on a network. IPFIX (Internet Protocol Flow Information Export) is based on a NetFlow implementation and serves as an industry standard for the export of flow information from network devices.

Quiz questions:
1. What is the term for tying individual events together to provide meaningful alerts? A. Event correlation B. Alert respondence C. Event mutuality D. Alert aggregation
2. You have been tasked with setting up a remote logging facility to send logs from various applications and network devices. Which of the following is the best choice? A. sFlow B. journalctl C. Syslog D. NetFlow
3. Which of the following provides information about other data? A. nxlog B. SNMP C. syslog-ng D. Metadata

Answer 1: A. Event correlation, often performed by security tools such as a SIEM systems, provides the ability to tie together individual log entries from across systems into meaningful and actionable alerts. Answers B, C, and D are incorrect.
Answer 2: C. A syslog server can listen for clients, thus acting as a remote logging facility. Answer B is incorrect as journalctl is only used to query journald. journald is a logging service, but it is not capable of logging to remote locations. Answers A and D are incorrect because these are both related to monitoring network flow packets.
Answer 3: D. Metadata is data that provides information about other data. Metadata is useful in regard to investigations, and practically everything digital contains metadata, including files and email. Answer A is incorrect because nxlog is used for centralized logging across various platforms and supports a myriad of different log types and formats. Answer B is incorrect because Simple Network Management Protocol (SNMP) is an application layer protocol whose purpose is to collect statistics from TCP/IP devices. Answer C is incorrect because syslog-ng is a log management solution that helps improve the performance of a SIEM system.