CompTIA Data+ DA0-001 Exam: Understanding Data Acquisition and Monetization

This guide covers Objective 2.1 (Explain data acquisition concepts) of the CompTIA Data+ exam and includes the following topics:
- Integration
- Data collection methods

This guide covers topics related to data acquisition and data monetization. It is important to understand the integration of data from both extract, transform, and load (ETL) and extract, load, and transform (ELT) perspectives. In addition, you need to understand that application programming interfaces (APIs) act as glue between disparate systems.

In this guide, you will also learn about data collection, including web scraping, public databases, API/web services, survey, sampling, and observation.

Integration
Before we get into the intricacies of data acquisition and monetization, it’s important to understand the basics of data mining.
Yes, data mining is a topic that sounds like mining minerals and ores—and really, it’s not very different from real-world mining of rare earth metals or gems. The main difference is that data mining is pertinent to data and happens (mostly) in the virtual world of computing, whereas the mining of metals and compounds is a physical-world process. After all, data is an extremely precious commodity in the virtual world—akin to diamonds and titanium in the physical world.
Data mining is the process of extracting usable information from huge amounts of raw data. Data mining analyzes patterns of data with the help of one or more software tools in order to drive informed decisions based on information mined from seemingly gibberish datasets.

Applications of data mining are diverse and are implemented in various fields, such as:
- Research and development
- Mining of rare earth metals
- Genetics research
- Cybernetics
- Marketing

In a business context, data mining allows, for example, salespeople to know about their clients in detail as well as to expand and execute effective sales strategies insightfully and optimally. It helps businesses move from randomly experimenting with sales strategies to establishing structured engagement with clients.
Data mining involves efficient collection of data, data processing, and data warehousing. To segment the data effectively and assess the likelihood of future events, data mining adopts complex mathematical algorithms.
Data mining is also referred as knowledge discovery in databases (KDD).
Now, before we dig deeper into data mining methods and the tools used for it, there’s a key term you need to understand: data monetization. Data is mined to help accelerate decision making and to get insights about what might happen and how to benefit from a trend or pattern. It is therefore possible to monetize data and get measurable economical upsides by investing in the process of data mining. Remember that businesses will invest in something if the activity offers return on investment (ROI), and data mining is one of many such activities—and it is an important one.

Some of the key characteritics of data mining processes involved in data mining are as follows:
- Predicting probable results
- Predicting patterns based on behavior and trend analysis
- Finding information to make decisions
- Working with very large databases and/or datasets for analysis
- Using powerful computing systems and large storage capacity to process and maintain information

Data Integration
Data integration involves combining business and technical processes for collating data from different sources into valuable and meaningful datasets. Data integration does not simply move data from point A to point B; it also makes data usable in the context of moving data from source systems to destination systems and then leveraging the information.
Contemporary data integration solutions integrate on-premises data integration tools with cloud-enabling apps to leverage data across both platforms. It is very common today to see hybrid data platforms with data sources on-premises and data processing happening in the cloud.
Data integration can happen in a number of ways, such as through ETL, ELT, and APIs.
As the times have changed, so have data integration solutions. Interestingly, cloud-based data integration solutions tend to be more user friendly than their on-premises counterparts; business users and IT staff do not need specialized skills to leverage cloud-based data integration solutions. With cloud systems, the configuration is typically straightforward, and the user interface is intuitive.

Extract, Transform, and Load (ETL)
As the name suggests, ETL tools enable data engineers to extract data from multiple source systems, transform the raw data into a more usable/workable dataset, and load the data into a storage system so that the end users can access meaningful data (with minimal noise) and use it to solve business problems. 5.1 illustrates this process.


The ETL Process

ETL tools are developed to save money and time as well as to eliminate the need for hand-coding (or other manual efforts) when new data warehouses are developed. For example, in a financial organization that stores and uses data about its clients in different departments and divisions, each department has client data stored and used in a unique way. For example, the accounting department stores the client data by account number, whereas the membership department stores the client data by member number. ETL tools can help the organization collect the data from all the varied sources and combine it into a standardized representation in a database or data warehouse.
ETL processes leverage a schema-on-write approach.
ETL tools extract information from heterogenous data sources (some of which may be legacy sources). They then transform data into a desired schema that is optimized for storing the information and performing analysis. After it is transformed, the data is synchronized and cleansed. Finally, these tools load the information into a data warehouse.

Following are the three stages of ETL:
- Extract: In this stage, the data is extracted from one or more source systems. This is often a complicated task because the information must be extracted appropriately in order to proceed further. Unless the right source systems are known and data is collated from these systems in a timely way, the outcomes can be underwhelming. These are some possible sources:
- SQL or NoSQL databases
- Flat or text files
- Customer relationship management (CRM) systems
- Enterprise relationship management (ERP) systems

Extraction includes parsing of extracted information and also verification of whether the extracted information achieves an expected structure or pattern. If data does not meet the expected structure, then it is partly or entirely rejected.
- Transform: The main aim of this stage is to convert the data into a single format such that it becomes usable by multiple applications. In this stage, data is cleansed and mapped to a specific schema. This process usually involves data staging with monitoring to ensure the quality and integrity of the data. In the event that there are changes or the data quality is not acceptable, the data can be repaired or discarded within the staging database before it goes to the loading stage. A series of business functions or rules are applied in the transform stage to extract the information into the required or desired schema. During the transformation phase, the following may occur:
- Cleansing and de-duplication of the data
- Validation and authentication of the data
- Mapping, translation, and/or summarization
- Data quality and integrity checks
- Formatting of data into desired data schema
- Load: In this stage, data is loaded into the final database or data warehouse from the staging database. This process may differ slightly, depending on the business requirements. During the load stage, the data can be structured according to the limitations, triggers, functions, uniqueness, mandatory fields, and referential integrity of the destination storage. This helps improve the overall data quality and performance in the ETL process.

During the ETL process, metadata is usually stored in a dedicated metadata repository where the users can retrieve, manipulate, or query the metadata.


The Detailed ETL Process

The basic steps of a real-life ETL cycle are as follows:

1. Initiate the ETL cycle.
2. Build data for reference.
3. Extract data from sources.
4. Validate the data.
5. Transform the data by performing operations such as cleansing, adopting rules for business, verifying for integration of the data, and creating disaggregates or aggregates.
6. Load data into stage tables, if used.
7. Create audit reports such as for conformity with rules for the business to assist to repair or identify the reason for failure.
8. Publish data to target tables.
9. Archive data.
10. Delete data as per business or regulatory requirements.

Extract, Load, and Transform (ELT)
As the name suggests, ELT is another process involved in data acquisition. While it might sound like ETL, it is a different process with different usage and execution processes. In the ELT process, there is no need for data staging, and most ELT tools leverage cloud-based data lakes or data warehouses for processing raw, unstructured, structured, and semi-structured data. From data sources, the raw data is moved into a destination system, such as a data warehouse or a data lake.


The ELT Process

Finally, the data is transformed at runtime (or on the fly) as required by the business, and data insights are pulled from transformed data on a visualization dashboard or in a report.
ELT processes leverage a schema-on-read approach.
ELT is typically leveraged during business hours, when users run analyses on the data collected from multiple sources in order to gain better insights. This requires the data to be transformed in near real time.
ELT is increasingly popular among data engineers and data scientists, as well as in context of cloud computing. Be sure to understand ELT for the CompTIA Data+ exam.

The three stages of ELT are as follows:
- Extract: During this stage, data is exported (or copied) from multiple data sources to a data lake or data warehouse. Data may be raw, semi-structured, unstructured, or structured, and the following types of data can be processed by ELT tools:
- Document and (flat) text files
- NoSQL and SQL data
- Emails
- Blog posts, articles, or web pages

- Load: After the extraction stage, the data is stored in a data lake or a data warehouse. It is quite usual for organizations to have a data loading process that is well defined, automated, and continuous. Any required business rules and data integrity checks can be run before the data is loaded into the data lake or warehouse.
- Transform: In this stage, the schema is applied to data before the analysis occurs. The functions executed in this stage are as follows:
- The data is cleansed, filtered, authenticated, and validated.
- Translations, analysis, calculations, or summaries are performed on the raw data. This function involves tasks like modifying column and row headers for consistency, changing units of measurement or currency, averaging or adding values, and editing text.
- Data is encrypted or masked in line with industry or government regulations.
- Data is formatted into joined tables or tables on the basis of the schema adopted.
-

The Detailed ELT Process


TABLE: Differences Between ETL and ELT Processes

Now, for the million-dollar question: Should your organization opt for ETL or ELT?
Ultimately, the business requirements and outcomes expected should guide the selection of ETL or ELT. For example, data scientists prefer ELT, as it involves collecting raw data and allows them to customize and transform the data based on their needs on the fly. They don’t need to focus on the structure of the data as the focus is on insights. For example, if there is a huge sale, data scientists can predict which items are performing better than others so more similar items can be put on sale.
On the other hand, a business with mostly batch data may use ETL for transactional data loading and analysis. In this case, the structure of data is much more important, and the outcomes do not differ very much over time. For example, at the end of each day, all records from online sales may be processed as a batch, and insights may be driven based on which products sold better than others during the day.

Delta Load
Delta load refers to the process of extracting only the delta—that is, the difference in the data compared to what was previously extracted—as part of the ETL process. It implies that the whole dataset will not be extracted from the table(s) but only the new information will be extracted and loaded to the target data store. In the context of ETL, a full load occurs when you load data for the first time (that is, when you are seeding the destination with initial data). Subsequently, a delta data load occurs when you are either loading changes to already loaded data or adding new transactions. To enable delta loading, it is important to determine which rows (or columns) in the table were already extracted and which dataset(s) is new or should be updated or added to existing datasets.

Application Programming Interfaces (APIs)/Web Services
Application programming interfaces (APIs) have become the new standard for system integration. Think of APIs as a bridge between newer, modern systems and disparate and older (possibly proprietary) systems. An API is collection of well-defined rules that provides details on how applications should interact with other applications.
An API provides a programmable interface for interacting with applications and infrastructure. With APIs you can:
- Create custom integrations between applications
- Create automation tools to simplify application provisioning
- Create a middleware layer to abstract one set of applications from other applications

Hence, APIs enable organizations to selectively share their applications in terms of data and functionality with internal stakeholders (such as developers and users) as well as external stakeholders (such as business partners, third-party developers, and vendors). Consider the example of a transport and logistics organization that has to integrate its order management, tracking, and other systems with vendors and suppliers. Instead of doing this integration on a system-by-system basis and creating dedicated plug-ins, the organization can opt for APIs and offer access based on role-based access control (RBAC) to only parts of the system, as required by the partners.
APIs can be based on multiple frameworks, standards, or protocols, such as representational state transfer (REST), Simple Object Access Protocol (SOAP), and Remote Procedure Call (RPC). Further, APIs can be classified on a systemic or functional basis—that is, based on systems for which they are designed, such as database APIs, web APIs, and remote APIs. APIs can also be classified as follows:
- Private: Only the organization developing the API has access to it.
- Public: Anyone has access to the API.
- Partner: The API is available to only a set of partners.

When working with applications and underlying infrastructure, APIs can be defined as northbound or southbound. A northbound API interacts with applications, and a southbound API interacts with underlying devices/infrastructure.
Let’s look at an example of a user trying to browse a web server and also interacting with rows in a database that is hosted behind the web server as a back end. 


An API Ecosystem

Based on 5.5, the high-level traffic and information flow is as follows:
A user web/application starts an API call for retrieving information; this is referred to as a request.
The request is intercepted by the API gateway, which proxies the API call to the web server. In this case, the web server is communicating with the back-end database using APIs as well.

The request may consist of the following methods:
- Get: Requests data from the source (for example, browsing web server)
- Put: Replaces data at the destination (for example, updating a database row)
- Post: Submits data to the destination (for example, sending login credentials)
- Delete: Removes data from the destination (for example, removing a row from a database)

In response to the request, the web server completes the required operation (according to the method used during the request) and returns a response to the user app via the API gateway.
With REST APIs, results can be published using status codes such as 2XX for success, 4XX for a client error, and 5XX for a server error.
In terms of web services, it is a simpler (and older) model that involves exchanging information between two machines on the Internet. While this process is still largely in use, it is being replaced by APIs as they do not need to broker connections for better control and security. A web service implies that there is a web server that runs on a virtual machine, physical server, or containers and listens for requests from other machines. When a request is received over a network, the web service responds with the requested resources, such as a Hypertext Markup Language (HTML) file, images, audio, or Extensible Markup Language (XML) files. 


Overview of the Web Services Process

Data Collection Methods
As established earlier, data is an invaluable currency, and organizations are increasingly collecting data for analysis. Data collection and analysis is important in any industry vertical, whether medical, marketing, or sales. For example, using data for accelerating product or service adoption in an organization matters a lot to modern enterprises. What is vital to understand is that data collection is not a random process; data collection occurs in an organized manner to ensure that the quality of data is high and the collection occurs in an ethical manner (as discussed later in this guide).
There are a number of ways that organizations can collect data, including from primary data sources (that is, direct data collection) and from secondary data sources (that is, indirect collection). With primary data collection, the data is collected firsthand from researcher or surveyor. With secondary data collection, it is the data which was previously gathered, processed, published, or analyzed. This data may be gathered from internal or external sources by investigators, researchers, or surveyors for carrying out statistical analysis and may be referred to as secondhand data.
The sections that follow cover the most commonly used data collection methods.

Web Scraping
The World Wide Web (WWW), also commonly known as the Internet, is a huge source of data, and because of the nature of the WWW, the data is mostly unstructured data. Unstructured data is difficult to collect because the file types, sizes, and formats can be varied.
Web scraping—also known as web data extraction or web harvesting—is a method used to extract data from websites and export it into a user-friendly format. While web scraping can be a manual process, more often than not, automated tools known as web scrapers are used for scraping web data; these tools are much more cost-effective and produce results faster than manual efforts. There are sites that offer web scraping as a service. For example, https://webscraper.io/ offers a web scraping tool—that is, an application that’s designed specifically to scrape relevant information from websites. A web scraping tool can leverage APIs and other methods to scrape data; in the process, they fetch unstructured data from web pages and change it to structured data as well as store it in data stores.

The web scraping process works as follows:
Identify the target websites or web pages and collect their Uniform Resource Locators (URLs) using a crawler.
Imitate a user browsing request to get the HTML code from the web pages.
Use a web scraper to try to find the interesting data in the HTML code and extract it.
Translate unstructured data into structured data (in a JSON or CSV file) for storage and analysis.



Web Scraping Process

A crawler (also known as a spider) is an automated tool that browses the targeted websites (which could be any website unless crawling parameters are well defined) on the Internet to index and search for content. The crawler goes through one or more specified websites to discover URLs and pass them on to the scraper. The scraper is a specialized application specifically designed to quickly extract data of interest from the crawled web pages. Web scrapers can find interesting data that should be extracted from an HTML file by leveraging HTML parsing libraries, CSS selectors, regular expressions (regex), or other methods. The data is then stored in an unstructured format such as a JavaScript Object Notation (JSON) or structured comma-separated values (CSV) files.
Why perform web scraping? Well, web scraping is seen as fundamental to the way some businesses operate, in that their decisions are based on the data being scraped from other sites. For example, in e-commerce, web scraping can help an organization understand the trends of a competitor and the prices as they’re set, in near real time. The most common applications of web scraping results are lead generation, academic research, marketing, and news.
Further, web scraping is an effective technique for collecting big data where collecting huge volumes of data is significant. In addition, search engines use web scraping in combination with web crawling for indexing the WWW in order to create huge volumes of searchable pages. Crawlers grab links so that they can determine where to store data in the record.

Public Databases
A public database is a compilation of data from previously distributed text presented in the public domain. It may contain data from online and offline sources such as literature, reports, textbooks, magazines, newspapers, articles, and published papers.

Surveys
Surveys have been used for many decades to collect data from targeted populations in order to understand patterns, demand, or trends so that businesses can align themselves better to consumers’ expectations. Survey data is typically collected from selected participants regarding a particular topic.

Several methods exist for collecting survey data and for performing statistical analysis on the data collected, including:
- Online surveys
- Paper surveys
- Telephone surveys
- Face-to-face, or in-person, surveys
 
Different communication media can be adopted for collecting opinions and feedback from survey respondents. The major factors that influence survey data are:
- How the interviewer will communicate with participants (such as offline or online)
- How the information on data being collected is presented to the participants
- The quality of data and the efficiency of the data collection process
-
The sections that follow describe the various survey methods.
Surveys in any form are an important topic as surveys help collate information in various formats. Be sure to understand the various ways in which surveys are useful to collect data.

Online Surveys
Online surveys are the most cost-effective surveys, given that their reachability is greater than can be achieved with other types of surveys, such as face-to-face surveys, telephone surveys, and paper surveys. Following are the highlights of online surveys:
- The investment needed for creating surveys and collecting the survey data is minimal compared with other methods. A good example would be a follow-up survey to an e-commerce order you recently placed.
- A researcher can ask more questions of the sample population in the same amount of time compared to telephone or face-to-face surveys.
- Online surveys are usually simple in their execution and take a small amount of time for participants to answer.
- Outcomes can be gathered in real time so that analysts can analyze the data and determine corrective measures.

- Online surveys are secure and safe to carry out. No in-person interaction is required, which is especially helpful during a pandemic. For example, during the COVID-19 pandemic, organizations shifted to contactless research and surveys.

Paper Surveys
Paper surveys are commonly used to collate information from respondents by leveraging paper and pen. They are adopted where tablets, computers, and laptops are either not allowed or feasible.
Paper surveys can be more expensive than online surveys as the surveys need to be printed, and someone needs to make the rounds to get respondents to respond to the survey. For example, a restaurant may use paper surveys to gather feedback from customers about the quality and appeal of its food.

Telephone Surveys
Telephone surveys require more investment than online surveys as inbound or outbound phone calls must be handled by real people. The cost of telephone surveys is a little higher than the cost of online surveys. Following are the highlights of telephone surveys:
- Communicating with participants via telephone requires less staffing and effort than communicating in person.
- The major disadvantage of carrying out telephone surveys is that it takes time to create a friendly environment with the respondent, where it is not possible to put a face to a name. Phone surveys occur in real time, and the respondent may not have time to answer questions.
For example, a research organization might carry out a telephone survey about the buying experience and motivation to invest in different brands.

Face-to-Face Surveys
In-person human interaction has historically been the best way to connect with people and build trust with emotions. Obtaining data from participants through face-to-face surveys has conventionally been much more successful than using the survey methods discussed earlier. When participants trust researchers and give honest feedback regarding the survey subject, the chances of collecting good-quality data increase.

Following are the highlights of face-to-face surveys:
- Surveyors can determine whether participants are comfortable with the survey questions and can help clarify if there’s any doubt.
- With face-to-face surveys, surveyors are likely to be aware of demographics, which gives them an advantage over online surveyors.

Post-Survey Actions
After survey data has been gathered, the information is analyzed to support the original intent of the research. Steps followed in analyzing survey data are as follows:
Understand the most common questions and the most common responses to those questions.
Filter acquired outcomes (such as by using cross-tabulation).
Evaluate the obtained information.
Make conclusions based on analysis to inform outcomes.

A number of data analysis techniques can be used to convert information to insights, based on the data obtained in surveys. These are as follows:
- Total unduplicated reach and frequency (TURF) analysis
- Strength, weakness, opportunity, and threat (SWOT) analysis
- MaxDiff analysis
- Gap analysis
- Cross-tabulation
- Conjoint analysis
- Trend analysis

Sampling
Sampling is a statistical analysis process that aims to collect data from a subdivision of a given population in order to gain insights about the whole population. For example, if you have been given the task of studying the market to determine the viability of a new product, instead of asking all possible consumers about their preference about the product, you will survey a sample of the population for input purposes, ensuring that the sample represents the whole population in terms of important factors such as age or sex. Their inputs would represent—statistically—the inputs that would have been gathered from a larger population.
Sampling is a complicated process, as the sample population and the inhabitants of the entire population can be very different in their ways of thinking and their responses.
 

There are various types of sampling methods, including the following:
- Random sampling: As the name suggests, this sampling method involves randomly choosing participants with no order or design—much like randomly selecting a raffle winner from ticket bowl.
- Systematic sampling: This sampling method follows set guidelines for better reliability in sampling. For example, a surveyor might consider only the first 10 out of every 20 respondents until 100 respondents have been selected.
- Clustered sampling: This sampling method looks at clusters, or subgroups, of inhabitants rather than at individuals.
- Convenience sampling: This sampling method leverages a readily available sample (that is, the easiest or closest respondents). For example, a surveyor might stand by a coffee shop, asking all customers about their preferences about a new gadget.
- Stratified sampling: This sampling method classifies individuals into subgroups of people who share similar characteristics or properties. For example, a surveyor may look only at people between 50 and 60 years of age and of either sex for a product that is suitable only for people in that age bracket.

Observation
Observation is one of the oldest methods of data collection, and it is used extensively in scientific research. It enables an observer or investigator to gather data about behavior or surroundings or individuals and then analyze the data in order to reach conclusions. Observation can be categorized based on the environment it is performed in, such as observing participants in a natural environment (participant-based observation), in a controlled environment (structured observation), or spontaneously with a phenomenon (spontaneous observation).
Using observation as a method for collecting data has a number of benefits, including:
- The observer does not need to have technical skills for data collection.
- Observation offers freedom in terms of describing the respondents’ activities and behaviors.