CompTIA CySA+ Cybersecurity Analyst Certification: Security Operations and Monitoring - Automation Concepts And Technologies

Objective 3.4  Compare and contrast automation concepts and technologies

Automation Concepts
The data that an organization produces every day is voluminous. Think about log data, transaction histories, network traffic captures, and so on. Considering all this data—likely terabytes of it daily from even a small organization—no one could hope to analyze or even view it all without help. Even scores of cybersecurity analysts could not get all of the work done in only a small-sized organization if they had to do everything manually; this would include log analysis, monitoring network traffic, risk analysis, threat monitoring, and all of the other detailed minutiae that infrastructure requires someone to watch over and act upon.
Fortunately, cybersecurity analysts do not have to do everything manually. Over the course of the years following the invention of the Internet and the development of sound cybersecurity engineering processes, IT and cybersecurity professionals have also built and honed their tools of the trade to make their work not only more efficient but more effective. These tools quickly handle the incredible volume of daily repetitive, tedious, and sometimes mind-numbing tasks that would take an analyst hours or even days to perform. This is especially advantageous for critical tasks that require precision and focus; human analysts may miss something important even in a simple pattern analysis task that can directly affect the security of the network; given the right set of criteria, automated methods of data manipulation and analysis don’t miss as much as a human might.

This module is focused on those automated methods, processes, and tools that cybersecurity analysts use to collect, ingest, aggregate, analyze, and make decisions on data. It is also focused on methods we use to perform repetitive but critical tasks to save time and make us more effective.

During this objective, we will discuss tools and methods such as workflow orchestration, scripting, interacting with software application programming interfaces (APIs), and automated malware management. We will also look at the processes that are more than just best business practices; they are critical to both the day-to-day and long-term infrastructure security management. These include data enrichment, threat feeds, machine learning, and automation protocols, as well as continuous integration and deployment.

Workflow Orchestration
Cybersecurity analysts often use a deep, multilayer cacophony of tools, processes, and methods for automating security tasks and managing security data. Many of these tools are stand-alone, and many of them do not necessarily allow themselves to be centrally managed. The ones that do may use different protocols or management interfaces; simply managing all these tools and processes that take care of task scheduling and performance or ingesting and analyzing data can be overwhelming. That’s where workflow orchestration comes in.

Workflow orchestration is the overall tactical and strategic management of all these automated (and sometimes manual) tools, methods, and processes.

Picture having to spend all day inputting data for data analysis software, scheduling automated tasks, and so on. This would not be a very good use of your time and really does not remove inefficiency; it simply moves the inefficiency from performing all these tasks manually to managing all the automated tasks. Workflow orchestration helps you to manage all of this.
There are certainly homegrown solutions you can use to integrate all your various security tools and scripts; there also open-source frameworks that allow this.

Some of the more robust commercial enterprise applications allow you to integrate compatible applications, such as security information and event management (SIEM) systems, intrusion detection/prevention systems, firewall management systems, and so on. None of them allow you to integrate every single tool you have; however, most use compatible protocols, messaging formats, languages, and other standards that can help integrate all these different tools. For example, most tools support XML input and output, so XML can be used to transfer data from one tool to another. Without a formal workflow management tool in place, there may be some programming involved to create an appropriate interface from an external tool if you have skilled programmers on staff who can facilitate this. Even with an enterprise-level workflow orchestration suite, there is still likely some setup or programming involved, but it will be much easier and efficient to implement.

Security Orchestration, Automation, and Response (SOAR)
Security orchestration, automation, and response (SOAR) is the name given to suites of tools dedicated to unifying your tools, processes, and methods used across the enterprise. These suites of tools can make it easier to ingrate all your efforts for collecting, aggregating, and analyzing data and then converting it to useful information used to make critical security decisions. SOAR primarily focuses on threat and vulnerability management, security operations, and incident response. Many of these tools are integrated as part of traditional network analytical tools, such as SIEM software.

Many SOAR implementations come in the form of layered security software modules. These tools enable you, often through agent-based endpoints and various threat and vulnerability feeds, to gather all this information into a customizable dashboard, providing real-time metrics and performance data as well as alerting you to potentially malicious event patterns. Many of these tools can also accept plug-ins from tools such as Python scripts, for example, and schedule and manage the execution of routine security tasks.

Exam tip: Workflow orchestration tools are used to provide overarching integration and management of all the disparate automated security tools, methods, and processes used by cybersecurity analysts.

Scripting
Scripting is a simple yet powerful way to automate tasks, particularly those that involve host or system-level tests. Scripting can also be used for network-wide tasks. Scripting at its basic level essentially involves taking commands you would normally run individually at the command line and including them in a file, called a script, so they can run sequentially and automatically. Occasionally, you must account for user input and either build that into the script or have a human being sitting there waiting for the script to ask for input.
Scripting can be quite simple or complex; most host-based scripts use the built-in operating system commands, but many use scripting languages such as Python, which must be installed on the system. Although scripts may be written and executed on the host, many scripts perform network-based tasks, such as collecting data, starting network backups, and so on. Scripts can be used to perform security tests as well, scheduled through the built-in operating system task scheduling mechanism (AT or cron, for instance) or as part of an overall management system that calls scripts on the fly when necessary, as certain conditions arise. You can have complex scripts that gather data from across the network and send it to a particular server for later analysis, and you can also use scripts to trigger a security alert if a predetermined condition occurs, such as a user being locked out of an account. Scripting can be useful when a graphical user interface for a particular security tool only provides limited or basic functionality. Scripting can allow an experienced security analyst to dig deeper and use more advanced tools than the graphical user interface permits, allowing more complex tasks to be performed.

Windows has various scripting facilities built in; for example, you can write and execute a simple command-line batch script consisting of operating system commands, or you can create powerful scripts using Windows built-in PowerShell feature. Linux also has built-in command-line scripting (called shell scripting) that can also be used to create both simple and complex scripts. For powerful, complex scripting needs that can be used to create cross-platform compatible scripts, many advanced cybersecurity analysts use scripting languages such as Python, which is essential to any analyst’s toolkit. There is also a multitude of other scripting languages, such as Perl, JavaScript, and VBScript, that are used to create applications, utilities, and interfaces across systems. Scripting is a critical skill that all cybersecurity analysts should learn.

Application Programming Interface (API) Integration
An application programming interface (API) provides a method for programmers to interface with an application’s functionality. Through APIs, developers can create additional functionality for applications, or interface one application with another application completely. APIs enable the exchange of information between disparate applications if the data is in a similar or translatable format. This includes executing different processes and functions in one application initiated by another.
API integration involves automating API calls between applications and converting them to code that each system can use. This can include authentication between applications if it is required. Many different tools can assist in automating API calls, such as Insomnia, Codegen, and Postman. Automating API call interfaces between applications lends itself to further consolidating the various security tools under one orchestration umbrella.

Exam tip: For this objective, you do not need to know the technical aspects of API programming; however, you should know how API integration is useful for automating security tasks.

Automated Malware Signature Creation
We discussed malware signature creation within Objective 3.2 and mentioned the popular YARA tool. YARA can automate the creation of malware signatures by creating rule-based detection signatures dynamically after analyzing strings found in malware files. Using tools such as YARA, along with scripts, as well as advanced methods such as machine learning (discussed in an upcoming section), cybersecurity personnel can automate the creation of malware signatures. This allows for malware signatures to be created based on the existing environment, as well as any external data the organization collects. This can also include threat intelligence, vendor malware information feeds, as well as other behavior analytics data from third-party sources.

Data Enrichment
Data enrichment is an important yet simple concept. Rather than only working with singular pieces of data, data enrichment refers to combining related data from other sources, both automated and manual, to give data context. This can help you see different dimensions of a piece of data. Some of this data enrichment may come from threat intelligence feeds, open-source information, various security vendors, government agencies, and even professional organizations. Much of this information is presented in formats that are easily ingestible by security tools.
Data aggregation can also help with the enrichment process. The more data you have put in context, particularly related data, the more data will make sense, and you can look at other aspects of it, other than the facts that stem from the data itself. It can help you understand why something happened, how it happened, and other details that might not be obvious from the data itself. Data enrichment can help humans analyze data and form better hypotheses regarding malicious or unusual events as well as historical and trend analysis.
Data enrichment is as much data aggregation and correlation as anything else. Let’s suppose that the piece of data you’re looking at is as simple as an IP address and domain name. Collecting data from sources other than your log files from your intrusion detection system, such as threat feed, open-source intelligence, and so on, can give you insight as to who owns the domain, if it has been seen before in other potentially malicious attacks, and so on. Data enrichment can help you put the piece of data in the proper context.

KEY TERM: Data enrichment means to give existing data derived from sources across the infrastructure context by adding more data such as that from threat intelligence feeds, security agencies, vendors, and so on. This can help with analysis and generating hypotheses.

Threat Feed Combination
There are various sources for threat data and intelligence. These can come from disparate sources, such as government agencies, vendor subscription services, and open sources. Because of the multitude of sources you have at your disposal, threat feeds can be overwhelming and sometimes confusing. That’s where developing a process to aggregate or combine threat feeds and distill out the relevant data is important. Many of these threat feeds conform to some of the standards we discussed in Objective 1.2, but much of it is unstructured data. Enter automation, once again, which can be used to sort through various feeds, search for patterns that you specify as interesting, and efficiently look for context that can be used to combine relevant information from the different threat feeds.

Machine Learning
Machine learning (ML) is a discipline of computer science that can be used to expand data analytics significantly beyond simple pattern searching or correlation. Starting with predetermined criteria, including conditions that could expand beyond the patterns of data you’re examining, machine learning can allow you to look past the obvious and explore avenues of data analytics you previously had not thought of. Intelligent software can extrapolate beyond its programmed pattern analysis and combine elements of behavior analysis with complex algorithms.

Machine learning systems use methods known as algorithms. Algorithms improve over time with new data and previously developed patterns and relationships. Therefore, the machine “learns” from the data it gets and its previous analysis.
Machine learning models improve performance and efficiency by adaptively learning not only from patterns (signatures) and behavior analysis but also from historical and trend analysis. This can enable machine learning software and systems to search through terabytes of data to locate potentially malicious events with greater accuracy. Data is key to machine learning. The more data the system receives, the better it is at searching for patterns of data, classifying it, and predicting outcomes of possible actions based on the activities of the patterns. Learning comes when the system can identify previously unknown patterns and relationships between data and sources. This makes it also able to detect anomalous behaviors as well.
Machine learning will not enable you to develop new hypotheses from data not already present. However, machine learning is especially useful when looking at the volumes of data produced by an organization and determining if there are potentially malicious activities going on, based on the relationships between data as well as the rules or patterns it is analyzing. Machine learning is also useful for things such as malware analysis, attack causality, and performance issues. It can be used to explain a historical analysis of data as well as predict potential trends. Machine learning is also good at recognizing patterns, such as network activity, facial recognition, and so on, if the algorithm can distinguish the data and analyze its behavior.


KEY TERM: Machine learning uses complex systems and intelligent software to ingest massive amounts of data, look for predetermined conditions and patterns, discover potential matches to those patterns, and extrapolate additional behaviors based on interactions between data and its patterns.

Use of Automation Protocols and Standards
Systems security professionals use a wide gamut of disparate systems to do their job; there are many standards, protocols, connection interfaces, and data formatting methods as well as markup, scripting, and programming languages that security tools and applications use, thus making it difficult sometimes to interface and integrate them and their data sources. Sometimes this creates issues with data formatting, quality, sufficiency, and transference to or from different security tools. This may require security professionals to build a bridge or interface of some type to transform data and send it to another application. Although there is a wide push to standardize tools and data, such as using common network protocols, markup languages, and so on, there is still a need to develop automation-specific protocols and languages. Fortunately, there is also an effort toward that goal as well. One such effort, known as Security Content Automation Protocol (SCAP), is promulgated by the National Institute of Standards and Technology (NIST).

Security Content Automation Protocol (SCAP)
NIST, in conjunction with other major organizations, such as Carnegie-Mellon University and the MITRE Corporation, as well as federally funded research and development centers (FFRDCs), has developed a common set of languages and formats for expressing security data to be exchanged between tools. Also, NIST has led the way in efforts to standardize data formatting and interfaces. These different automated languages include those for expressing vulnerabilities, threats, assets, mitigations, and so on.
One of the major automation protocols to come from NIST over the past few years is the Security Content Automation Protocol, or SCAP. This is more of a framework than a protocol itself; it is used to assist in the overall risk management of systems by providing a consistent, open data format that can be used across different security tools and platforms.

 Most major security tools, including vulnerability scanners and so on, are SCAP compatible. NIST Special Publication (SP) 800-126, Revision 3, is the guiding document for SCAP. 

SCAP Specifications and Categories ( the 12 standards in the five SCAP categories)


Exam tip: You should be familiar with the various SCAP languages and formats as well as their purposes for the exam.

Automating Software Integration, Delivery, and Deployment
Previously, in Objective 2.2, we discussed the principle of DevSecOps. DevSecOps was born out of necessity, due to problems inherent in the separation of the development, operations, and security communities. Prior to the integration of these three pieces of the software development process, software was often developed in a vacuum, with no input from the operational users for whom the software was intended or cybersecurity personnel. This sometimes resulted in software that not only did not meet its intended requirements or functions but also was not secure. With the merger of these three disparate communities into DevSecOps, the development of software has become more unified, and when properly implemented is able to produce secure, functional software that meets user requirements. Like all other efforts involving mass amounts of data, analytics, and security service delivery, the automation paradigm can be applied to software integration, delivery, and deployment.

Continuous Integration
Continuous integration refers to the ability to constantly produce quality software as it is being developed. In other words, changes to the software are made and rapidly integrated into the production code. Note that this does this doesn’t eliminate the need for testing, but it does give the team the ability to rapidly move forward in the development cycle, from the necessity to integrate changes to those changes becoming part of production code. Continuous integration lends itself to automation since there are tools that can integrate and deliver code from the test environment to the production environment. Even with automated means, this does not take the human out of the loop as a decision maker in ensuring that the code is ready to be put into production.

Continuous Delivery and Deployment
Continuous delivery is the process of moving production code from the development environment into the production environment. This is after the integration process has taken place. This process can be automated, including the security processes that need to follow it (vulnerability testing, for example). Again, there still a human involved in the decision-making process to move the code from development to production. Automating this process could be something as simple as a script that moves the integrated code from a developer file share to a file share that is accessible by user applications or processes. Once the newly integrated code has been released from the developer and delivered into the user space, other automated processes, such as security testing, can take place before the software is actually deployed to the users.
Continuous deployment is the third step in this process; in this part of the automated software process, after software has been integrated and delivered to the central delivery point, it can be deployed out to the users through whatever means the organization chooses. Again, there are automated processes to do this; some are built in to software management applications and deployment points, such as app stores, and others are simple scripts that can be created to deploy software. Software can also be automatically deployed as part of a policy, such as the software policies created in Active Directory, or as part of user-directed action.

REVIEW
Objective 3.4: Compare and contrast automation concepts and technologies In this objective, we discussed automation principles, methods, processes, and tools. These are necessary to handle the massive quantities of data that security analysts must deal with on a daily basis as well as to perform the multitude of tasks required to collect, analyze, aggregate, and use the data that’s produced by an organization.
First, we discussed overall management of all these automated processes and tools, and how not all of them lend themselves to centralized management. Workflow orchestration is one method of doing that and can be implemented using a variety of commercial enterprise tools or individual tools such as scripting. We also looked at security orchestration, automation, and response (SOAR) systems and how they can provide centralized management for automated processes.
Scripting is a critical skill that cybersecurity analysts should develop to automate both small and large tasks. Scripts can be very simple or complex, using built-in operating system commands or complex programming languages.

Application programming interfaces are used to connect applications together and interface them for data exchange or task performance integration. This allows security personnel to create a more unified management approach to multiple disparate applications. We also discussed automated malware signature creation, using tools such as YARA and scripts to search for patterns in existing files that may contain malware. Automation can take on much of the load in searching for and analyzing those patterns.
Data enrichment is essentially adding additional data and context to collected data from systems. This additional data context can help further analyze information to discern threats, unusual events, and undesirable trends in the infrastructure. This context can come from threat intelligence feeds, behavioral analysis, and so on.

Machine learning is a powerful tool used to ingest large quantities of data, look for patterns of behavior, and use complex algorithms to discover and extrapolate potential malicious behaviors in the infrastructure.
There are several efforts to produce automation protocols used to exchange data in standardized, common formats between applications. NIST has developed several structured language and reporting formats for identifying assets, threats, and vulnerabilities and expressing other security data for exchange between security tools.
Automation can also be applied to secure software integration, delivery, and deployment. Tools can be used to seamlessly integrate new code into existing production code, deliver it for security testing, and finally deploy it out to the users. Automated methods of doing this include scripting, software management tools, and even software policies built in to enterprise-level services such as Active Directory.