CompTIA CASP+ CAS-004 Certification: A Simple Guide To Enterprise Risk Measurement and Metrics

Securing an organization should be assigned the highest priority, yet the business and senior management need to be convinced to allocate budget and resources for ensuring top-notch security. Security heads need to justify implementing security controls or buying new security technologies.

After the security controls have been implemented, this guide presents the review and assessment of the effectiveness of the risk controls by gathering and analyzing the risk metrics. The results are further interpreted for future trends, existing security levels, and trends against industry standards and baselines.
- Effective security control review
- Create risk metrics
- KPI, RCO, TCO
- Compare security baselines
- Anticipate future needs
- Risk review of controls
- Analyze metrics
- Benchmarks
- Baselines
- Data analysis and interpretation
- Judgmental solutions

Securing an enterprise is very important. Security should be a top priority for any organization, but often it can be difficult to convince the senior management to provide the funds for the security endeavors you wish to use. As a security professional, you need to provide justification for any security technology and control you want to implement.

Structure
- Review the effectiveness of existing security controls
- Reverse engineer/deconstruct existing solutions
- Create, collect, and analyze metrics
- Prototype and test multiple solutions
- Create benchmarks and compare them to baselines
- Analyze and interpret trend data to anticipate cyber defense needs
- Analyze security solution metrics and attributes to ensure that they meet business needs
- Use judgment to solve problems where the most secure solution is not feasible

Objectives
This guide explains the effectiveness of security controls, gap analysis, and after-actions. The use of reverse engineering and deconstructing existing controls helps gather sensitive information about the organization’s controls as well as creating, gathering, and analyzing metrics including KPIs and KRIs. This guide explains solutions to decide upon the security controls to deploy for securing the organization. As a CSAP security professional, you need to provide justification for any security technology and control you want to implement.

Review effectiveness of existing security controls
Organizations should evaluate the efficacy of current security procedures on a regular basis. All areas of security, including security training, device configuration (router, firewall, IDS, IPS, and so on), and policies and procedures, should be reviewed by security specialists. Vulnerability and penetration testing should also be performed. These audits must be carried out at least once a year.

A review of the effectiveness of security controls should include answering the following questions:
- Which security controls are we using?
- How can these controls be improved?
- Are these controls necessary?
- Have any new issues arisen?
- Which security controls can be deployed to address the new issues?
To aid in the review of existing security controls, security administrators should perform a gap analysis and document the lessons learned in an after-action report.

Gap analysis
An information security gap study compares the security program of a company to industry best practices. Security experts can identify vulnerabilities and dangers by comparing these recommended practices to real practices. \

An information security gap analysis includes the following five steps:
Step 1: Select an industry-standard framework.
Step 2: Examine the individuals and procedures involved. Collect information on the IT environment, application inventories, organizational charts, rules and practices, and other important elements.
Step 3: Collect information and technology. This stage assists a company in determining how well its present security program functions inside the technological architecture. Comparing best practice controls or relevant requirements to organizational controls; sampling network devices, servers, and applications to validate gaps and weaknesses; reviewing automated security controls; and reviewing incident response processes, communications protocols, and log files are all examples of this.
Step 4: Analyze/compare the information you’ve acquired. This step entails using the information gathered to conduct an in-depth analysis of the organization’s security program, as well as correlating the findings and results across all factors to create a clear and concise picture of the organization’s IT security profile, including strengths and areas for improvement.
Step 5: Gap analysis is finally performed. Gap analysis is a complex, in-depth procedure that necessitates a full understanding of security best practices as well as a complete understanding of security threats, controls, and operational difficulties. While doing a gap analysis does not ensure 100 percent security, it does go a long way toward ensuring that the organization’s network, staff, and security policies are solid, effective, and cost-effective.

The figure below illustrates the Gap Analysis steps:


Figure: Gap Analysis Steps

Lessons learned and after-action reports
When a problem emerges, security experts are generally focused on addressing the problem, implementing a new security control, or upgrading an existing security control. The lessons learned/after-action evaluation should be filed after the immediate crisis has passed. Personnel describes the issue facts, the cause of the issue, why the issue happened, feasible strategies to prevent the issue in the future, and improvement ideas in case the issue arises again in this report. Anyone who was engaged in finding or fixing the problem should be included in the review. Because specifics are sometimes lost over time, reviews should be held as soon as feasible after the issue has been resolved.
It’s ideal to format the official review document such that it follows the incident in chronological order. The investigation should include as many details regarding the occurrence as feasible. Remember that lessons learned/follow-up reviews may be used for any major organizational project, such as operating system updates, new server deployments, firewall upgrades, and so on.

Reverse engineer/deconstruct existing solutions
An organization’s security measures are only effective until the hacker figures out how to breach or bypass a control. As a result, a security professional’s ability to think like a hacker and reverse engineer or disassemble the existing security systems is critical. Each security solution should be examined independently by a security expert. When examining each solution, you should consider what the security solution performs, which systems it is supposed to protect, how the solution affects the company, and what information it discloses about itself. Keep in mind that the goal of reverse engineering is to learn as much as possible about your company in order to find a method to get into it. Physical entry to the building is sometimes overlooked by security specialists. Physical security controls, on the other hand, are just as critical as any other control. If an attacker can enter your facility and connect a rogue access point or protocol analyzer to the enterprise, it doesn’t matter how many security safeguards you have in place.

Creation, collection, and analysis of metrics
Metrics should be checked on a regular basis. Furthermore, measurements should be examined as quickly as possible after they are gathered to see if any modifications are required. Metrics that are properly created, collected, and analyzed, help a company forecast future demands far before an issue emerges. The organization’s security budget is prepared by the chief security officer (CSO) or another designated high-level management, who also defines the security metrics and reports on the security program’s efficacy. This officer must collaborate with subject matter experts (SMEs) to ensure that all security expenses, including development, testing, implementation, maintenance, staff, and equipment, are accounted for. The budgeting process necessitates a thorough analysis of all risks and guarantees that the most cost-effective security initiatives are performed. Long-term and strategic projects, which take more than 12 to 18 months to complete, require additional resources and finance. Both short- and long-term patterns may be found in security measurements. A security expert can calculate the daily workload by collecting these variables and comparing them on a day-to-day basis. The trends that emerge when the indicators are evaluated over a longer period of time might help determine future security programs and expenditures.
Who will collect the measurements, which metrics will be gathered, when the metrics will be collected, and what thresholds will trigger corrective measures should all be specified in the procedures. For assistance in building metrics rules and processes, security professionals could examine information security governance frameworks, notably ISO/IEC 27004 and NIST 800-55. Metrics, on the other hand, aren’t simply for live events. You may also use the generated data to examine the effects of security policies in a virtual environment that simulates the real world. Then you may utilize the simulated data to see if the security controls should be implemented in the real world.
Assume a security administrator is attempting to establish a body of knowledge to allow heuristic and behavior-based security event monitoring of global network operations. The instrumentation is chosen to enable network monitoring and measurement. Modeling the network in a series of virtual machines (VMs), implementing systems to record comprehensive metrics, running a large volume of simulated data through the model, recording and analyzing results, and documenting expected future behavior is the best methodology to use in establishing this baseline. The security administrator would be able to determine how the new monitoring would operate using this complete way.
Although the security team should examine metrics on a daily basis, a third-party study of the metrics on a regular basis may assure the security metrics’ accuracy and efficacy by confirming the internal team’s results. The data from the third party should subsequently be used to improve the security program and security metrics procedure. The two types of metrics that are generated, gathered, and assessed are key performance indicators (KPIs) and key risk indicators (KRIs).

Information Security Forum (ISF) recommends the following 14-step approach to KPIs and KRIs to support informed decision making:
Step 1: Understand the business context.
Step 2: Identify audiences and collaborators.
Step 3: Determine common interests.
Step 4: Identify the key information security priorities.
Step 5: Design KPI/KRI combinations.
Step 6: Test and confirm KPI/KRI combinations.
Step 7: Gather data.
Step 8: Produce and calibrate KPI/KRI combinations.
Step 9: Interpret KPI/KRI combinations to develop insights.
Step 10: Agree to conclusions, proposals, and recommendations.
Step 11: Produce reports and presentations.
Step 12: Prepare to present and distribute reports.
Step 13: Present and agree on the next steps.
Step 14: Develop learning and improvement plans.

Security experts must lead their organization in monitoring KPIs and KRIs using this method. A performance indicator is a measure that tells you how well your company is performing. It instructs you on what to do and how to proceed. Measures, which are observed values at a certain point in time, are used to create metrics. Metrics are ratios, averages, percentages, or rates produced from measures, whereas measures are raw numbers and data points.

KPIs
KPIs monitor factors that are directly related to individual actions or activities, rather than the end outcome. Profit, expenses, and the number of accounts should not be utilized as key performance indicators (KPIs). They are the consequence of a variety of activities; hence, they do not provide specific measures to perform.

KPIs that organizations need to capture include the following:
- Increase or decrease in reported incidents
- Number of large and small security incidents
- Cost per incident
- Amount of time for incident resolution
- Downtime during an incident

The figure below illustrates a sample KPI report:


Figure: Sample KPI Report

Let’s look at an example. Suppose an organization’s IT department reported a significant decrease in the reported incidents over the past quarter. Some questions that the management may need to look into include the following:
- Were new security controls put into place during the quarter that possibly caused this significant decrease?
- Was there an actual decrease in incidents or just a failure to discover or report incidents?
- What are the operational differences (for example, system upgrades, new tools, heavily attacked systems that have been patched, removed, or replaced) between the last quarterly report and this quarterly report?

KRIs are used in management to indicate how risky an activity is or how likely a risk is to occur. Organizations use them as early signals that particular risks may occur. KRIs that organizations need to capture include the following:
- Acceleration of high-severity events: Are more severe events showing up on your systems in a shorter amount of time?
- Handle time: How long does it take for you to identify a threat-pattern change and eliminate the cause of that threat?
- Attack surface area: How many hosts are involved in a security event? How many hosts are included in an attack?

The figure below illustrates a sample KPI Risk Report:


Figure: Sample KPI Risk Report

Let’s say a company is concerned about its security awareness training. Examining the pass/fail metrics for the security awareness training is a KRI for this. If the failure rate is high, the organization’s training practices must be improved, particularly the amount of time spent on training each year and the employee engagement index for the training. The pass/fail rate for security awareness training will be directly affected by the amount of time spent training and the amount of training delivered to personnel. In this case, the company may opt to require additional security awareness training for employees.

Prototype and test multiple solutions
Once a security specialist has determined that a device or technology has a definite problem, he or she should consider viable remedies. Hardware improvements, new device or technology purchases, and configuration adjustments are all possible remedies. The security professional should then prototype or test the solutions. Any prototyping or testing should preferably be done in a lab environment to evaluate the impact of any deployed solution. Prototypes also assist in ensuring that the tested solutions are satisfactory to the company before they are launched into production. Virtualization technologies have made creating and testing solutions in a virtual ‘live’ environment a lot easier. Make sure that any testing is done in a vacuum, with no other solutions implemented, to ensure that the impacts of that particular solution are completely understood. After you’ve figured out what each solution does, you may prototype or test various solutions to see whether it’s better to apply numerous solutions to your company’s problem.
Let’s say you uncover a web server that isn’t performing well. Deploying a second web server and putting both servers in a load-balancing environment is one method being examined. Upgrades to the hard disc and RAM of the concerned server might also be an option. Of course, an even better method would be to improve the original web server, deploy a second web server, and load-balance both servers. Budget restrictions, on the other hand, frequently hinder the adoption of many solutions. Testing may demonstrate that upgrading the web server’s hardware is sufficient. A hardware upgrade may be the best short-term answer until the funds for deploying a second web server becomes available. You may test the solution in the live environment once you’ve prototyped or tested it in the lab and cut down your solution options. Keep in mind that such solutions are generally best implemented during times of low traffic. Always make a complete backup of the device you’re upgrading before proceeding with the upgrade.
Also, keep in mind that monitoring performance and recording baselines and benchmarks will have an impact on the performance of the systems being watched. It’s critical to capture both a baseline and a benchmark at the right moment. When a system has been properly configured and upgraded, baselines should be taken. In addition, rather than a day or an hour, baselines should be measured over a longer period of time, such as a week or a month. When new baselines are created, they should be compared to the existing baselines. It may be required to adopt new baselines based on the most recent data at that time.

Create benchmarks and compare baselines
A baseline is a predetermined and documented reference point that will be utilized as a future benchmark. It’s important to acquire baselines, but it’s even more important to utilize baselines to analyze the security situation. Even the most detailed baselines are useless unless they are used. However, baselines will not help you if you don’t have current standards to compare to. A benchmark is a point of reference that captures the same data as a baseline and may even be used as a new baseline, if necessary. It is then utilized for comparison. A benchmark is compared to the baseline to check whether there are any security or performance flaws.
The sales team’s visit on Thursday explains the rise in authentication volume. On the other side, if you detect an increase in VPN traffic on Thursdays, you should be concerned since the sales staff will not be utilizing the VPN because they will be at the office. Understanding baselines and benchmarks also entail comprehending thresholds, which ensure that security risks do not escalate beyond a certain level. If the system administrators must be notified before a security event occurs, the ideal technique is to arrange the program to send an alert, alarm, or email message when certain occurrences reach the threshold. To ensure that they can appropriately notice when potential concerns emerge, security experts should collect baselines at different times of the day and on different days of the week. Furthermore, security experts must ensure that benchmarks are being compared to the correct baseline. Comparing a Monday 9 a.m. benchmark to a Saturday 9 a.m. baseline may not allow you to adequately analyze the issue. You should design a feasible remedy to every issue you uncover once you’ve identified the trouble areas.

Analyze trends and data
Analyzing and understanding trend data to predict cyber protection needs is a crucial step in safeguarding a company. Security specialists should be able to predict where and when defenses will need to be strengthened using trend data.
Let’s say you discover that the user accounts are getting locked out at an increasing rate over time. Several users claim that they are not responsible for their accounts being locked out. You think a hacker has gotten a list of user account names after analyzing the server and audit logs. Furthermore, you learn that the attacker is attempting to connect from the same IP or MAC address over and over. You may wish to adjust the firewall that safeguards your network to prohibit any connections from the attacker’s IP or MAC address after the analysis is complete. Changing all usernames is another viable security measure. Changing the user account names, on the other hand, might have ramifications for other services, such as email. As a result, the company could be ready to disregard the possibility that an attacker knows all the user account names.
Let’s have a look at a more complicated scenario. Assume a security administrator has discovered a slew of network issues plaguing the proxy server. The administrator observes that the firewall is being attacked with multiple web assaults at the same time that the network difficulties are occurring when analyzing the logs. Deploying a protocol analyzer on the switch span port, adjusting the external-facing IPS, reconfiguring the firewall ACLs to block unnecessary ports, verifying that the proxy server is configured correctly and hardened, and continuing to monitor the network is the most effective way to conduct an in-depth problem assessment and remediation.
It’s critical to keep track of such patterns so that the right security measures may be implemented before they turn into genuine concerns. Furthermore, tracking these trends might help you predict resource requirements before they become urgent. For example, if you detect that web server traffic is rising at a specific pace each month, you may plan for upgrades before the traffic reaches a point where the server becomes obsolete and can no longer process client requests.

Analyze security solution metrics and attributes
Security solutions are used to keep a company safe. When security experts install security solutions, they must first define a specific business requirement that the solution addresses. Performance, latency, scalability, capability, usability, maintainability, availability, recoverability, and cost/benefit analysis are the essential business needs to grasp for the CASP test.

Performance
The way in which or the efficiency with which a product or technology reacts or performs its intended goal is referred to as performance. The performance level that should be maintained on each device and throughout the enterprise as a whole should be determined by the company. Any security solutions that are installed should meet the performance criteria that have been defined. Performance criteria should consider both present and future requirements. For example, if a company wants to install an authentication server, the solution it chooses should meet the company’s present authentication needs as well as any future authentication requirements. Deploying a solution that gives even higher performance than required will allow the solution to be used for longer than expected.

Latency
Latency refers to the time it takes for network data to be processed. A low-latency network connection has relatively short delay periods, whereas a high-latency network connection has very large delays. Many security measures have the potential to increase latency. Routers, for example, take a certain amount of time to analyze and forward data. Configuring extra rules on a router often increases latency, consequently resulting in lengthier delays. Because of the detrimental consequences on network latency, an organization may opt not to use certain security measures. Auditing is an excellent example of a security solution that has a negative impact on latency and performance. When auditing is enabled, it logs certain activities as they happen. The latency and performance may be affected by the recording of these actions.

The figure below  illustrates network latency:


Figure: Network Latency

Scalability
Scalability is a feature of a device or security solution that specifies its capacity to cope with and function under a growing workload. Time factors are commonly used to describe scalability. In order to determine scalability, it is necessary to assess the existing and future demands. Scalability also refers to a system’s capacity to expand in response to the changing requirements.

The figure below illustrates the non-scalable setup:



Figure: Non-Scalable Setup

To improve performance, a scalable system can be enlarged, load-balanced, or clustered. Let’s say a company has to set up a new web server. An older system is found that can be converted to function as the new web server by a systems administrator. Following an assessment of the organization’s requirements, it is concluded that the web server will meet the organization’s present requirements. It will, however, be unable to meet the estimated demands in six months. If the expenses of the upgrade are not prohibitive, upgrading the server to boost scalability may be an alternative. The cost of the update and the new scalability value should be compared to the cost and scalability of a completely new system.

The figure below illustrates scalable setup:


Figure: Scalable Setup

Capability
The activity that a solution is capable of performing is referred to as its capacity. An intrusion detection system (IDS), for example, detects invasions, whereas an intrusion prevention system (IPS) prevents them. The mechanism by which a solution performs its functions, as well as any solution capabilities that the organization does not require, should be understood. Often, security solutions come with more features at a higher cost.

Usability
Making a security system or device easier to use and aligning it more closely to organizational objectives and requirements are referred to as usability. It’s critical to ensure that your organization’s employees can adopt and manage a new security system. When calculating return on investment (ROI) and total cost of ownership, all staff training expenditures must be included in the solution’s costs (TCO). Because of their usability, even the finest security systems may be eliminated as options.

Maintainability
The frequency with which a security system or device must be updated, as well as the length of time it takes to do so, is referred to as maintainability. Patching, cleaning logs, and updating software are all part of this process. When assessing maintainability, an organization should examine how much maintenance is necessary, how long it takes to complete, and how frequently maintenance is typically performed. Any projected future changes should also be factored into the maintenance concerns.

Availability
The length of time a computer system is available for usage, expressed as a percentage, is known as availability. The words maximum acceptable downtime (MTD), mean time to repair (MTTR), and mean time between failures (MTBF) are frequently used when determining availability, as presented below. You must be able to determine when new devices or technologies are being installed to boost data availability in order to pass the CASP test. Consider the following scenario:
Assume a small business uses a single host to host many virtualized client servers. To form a cluster, the organization is contemplating adding a new host. Although the new host’s hardware and operating system will differ from the original, the underlying virtualization technology will remain compatible. A shared iSCSI storage solution will be used by both hosts. Customers’ data will be more accessible, thanks to the iSCSI storage solution. The best way to assess availability is to look at the component of the security system that is most likely to fail. Knowing how long a solution will be unavailable, how long it will take to fix it, and how long it will take between failures are all critical factors in evaluating availability.

The figure below illustrates the system downtime availability:


Figure: System Downtime Availability

Recoverability
The likelihood that a failed security solution or device may be restored to its normal operational state within a certain time frame utilizing the defined practices and procedures is known as recoverability. The phrases recovery time objective (RTO), work recovery time (WRT), and recovery point goal (RPO) are frequently used when assessing recoverability. Researching the activities that will need to be conducted if a partial or complete recovery of the security solution or device is necessary is the best way to assess recoverability. When deciding between different security systems or devices, knowing how long it would take to recover is critical.

Cost/benefit analysis
Before introducing any security solutions to the organization, a cost-benefit analysis is conducted (Figure 5.8). This form of analysis compares the expenses of implementing a solution with the advantages that will be realized as a result of its implementation. For the most part, an organization should only deploy a solution if the advantages of doing so outweigh the expenses of doing so.

The figure below illustrates the cost-benefit analysis:



Figure: Cost-Benefit Analysis

Cost-benefit analysis is a key decision-making tool that helps determine whether a planned action or expenditure is literally worth the price. The analysis can be used to help decide almost any course of action, but its most common use is to decide whether to proceed with a major expenditure. Since it’s based on adding positive factors and subtracting negative ones to get a net result, it is also known as ‘running the numbers’.

The figure below illustrates the project-wise cost-benefit analysis:


Figure: Project wise Cost-Benefit Analysis

ROI
Return on investment (ROI) refers to the money gained or lost after an organization makes an investment. ROI is a necessary metric for evaluating security investments, as shown in Figure 5.10:


Figure: Return on Investment

TCO
The total cost of ownership (TCO) measures the overall costs associated with securing the organization, including insurance premiums, finance costs, administrative costs, and any losses incurred. This value should be compared to the overall company revenues and asset base.

The  figure below illustrates the TCO:


Figure: Total Cost of Ownership

Judgment to solve problems
You will frequently be asked for your input as a security professional. There is no actual right or wrong answer in these situations, and you will have to apply your judgment to tackle complex challenges when the most secure option is not practicable or if there is no optimum solution. The greatest thing you can do in this situation is to conduct a study. Use all of the resources at your disposal to learn more about the issue, including visiting vendor websites, polling your peers, and getting third-party comparative studies. Understanding why the most secure solution isn’t possible will help you choose a different one. Due to cost, time, or scope restrictions, the most secure solution may not be possible. Whatever the limitation, security experts must assist in the development of solutions to minimize the problem. As your expertise and knowledge grow, you’ll be better equipped to make these decisions based on that experience and knowledge while still conducting some research. Making excellent judgments requires information. Pose questions and receive responses. Then balance each of your responses to evaluate any solutions you’ve found. You’ll have to make a decision and live with it in the end. Making an informed decision, on the other hand, is always the better option.

Conclusion
This guide explains the review of effectiveness of security controls to determine new control deployment or deconstruct existing solutions to simulate attackers. Collection and analysis of metrics to use for determining which security control needs to be deployed are also discussed in this guide. Interpreting data to anticipate defense needs and take decisions is also discussed.