Intro to Marketing Research: Quantitative Research - Pre-testing of Questionnaires, Cognitive Interviews Pilot Surveys

What It Is

Pre-testing of questionnaires, also known as cognitive interviews or pilot surveys, is a method used to evaluate the effectiveness and clarity of survey questions before administering them to a larger sample. This process involves testing the questionnaire on a small group of respondents to identify any issues with question comprehension, response accuracy, or survey flow. A notable example of pre-testing is the 2010 US Census, where the US Census Bureau conducted extensive cognitive interviews to refine the questionnaire and improve data quality. This matters for marketing decision-making as it ensures that survey results are reliable and accurate, which is crucial for making informed business decisions.

Key Terms & Concepts

• Cognitive Interview: A method of evaluating survey questions by asking respondents to think aloud while completing the questionnaire.
  • Example: The US Census Bureau used cognitive interviews to refine the 2010 US Census questionnaire.
• Pilot Survey: A small-scale survey conducted to test the feasibility and effectiveness of a larger survey.
  • Example: A company may conduct a pilot survey to test a new product feature before launching it to a wider audience.
• Questionnaire Pre-testing: The process of evaluating survey questions before administering them to a larger sample.
  • Example: A market research firm may conduct questionnaire pre-testing to ensure that survey questions are clear and accurate.
• Survey Flow: The order and sequence of questions in a survey.
  • Example: A survey may have a poor survey flow if questions are too long or complex, leading to respondent fatigue.
• Question Comprehension: The ability of respondents to understand survey questions.
  • Example: A survey may have poor question comprehension if respondents are unsure what a question is asking.
• Response Accuracy: The accuracy of respondent answers to survey questions.
  • Example: A survey may have poor response accuracy if respondents are not answering questions truthfully.
• Reliability: The consistency of survey results over time.
  • Example: A survey may have high reliability if respondents are answering questions consistently over multiple administrations.
• Validity: The accuracy of survey results in measuring what they are intended to measure.
  • Example: A survey may have high validity if it is accurately measuring customer satisfaction.
• Exploratory Research: Research designed to explore a new topic or issue.
  • Example: A company may conduct exploratory research to identify new market opportunities.
• Descriptive Research: Research designed to describe a population or phenomenon.
  • Example: A company may conduct descriptive research to describe customer demographics.
• Cronbach’s Alpha: A statistical measure of reliability.
  • Formula: Cronbach’s Alpha = (k / (k - 1)) * (1 - (^2_x / ?^2_total))
  • Where k is the number of items, ?^2_x is the variance of each item, and ?^2_total is the total variance.
• Sample Size: The number of respondents in a survey.
  • Example: A survey may require a sample size of 1,000 respondents to achieve reliable results.
• Type I Error: The error of rejecting a true null hypothesis.
  • Example: A survey may have a Type I error if it incorrectly identifies a significant difference between groups.
• Type II Error: The error of failing to reject a false null hypothesis.
  • Example: A survey may have a Type II error if it fails to identify a significant difference between groups.

Common Misunderstandings

• Misunderstanding: Pre-testing is only necessary for large-scale surveys.
• Correction: Pre-testing is essential for all surveys, regardless of size, to ensure that questions are clear and accurate.
• Misunderstanding: Cognitive interviews are only used for survey questions.
• Correction: Cognitive interviews can be used for any type of survey question, including rating scales and open-ended questions.
• Misunderstanding: Pilot surveys are only used for testing survey flow.
• Correction: Pilot surveys can be used to test survey flow, question comprehension, and response accuracy.

Quick Application / Identification

Scenario: A company is launching a new product and wants to conduct a survey to measure customer satisfaction. The survey has 10 questions, including a rating scale and an open-ended question. What type of pre-testing should the company conduct to ensure that the survey questions are clear and accurate?

Answer: Cognitive interviews. Explanation: Cognitive interviews are necessary to evaluate the clarity and accuracy of survey questions, including rating scales and open-ended questions.

Last-Minute Revision

• A survey with a sample size of 100 respondents may not be reliable.
• Cronbach’s Alpha is a measure of reliability, not validity.
• Exploratory research is designed to describe a population or phenomenon.
• Descriptive research is designed to explore a new topic or issue.
• Type I error is the error of failing to reject a false null hypothesis.
• Type II error is the error of rejecting a true null hypothesis.
• A survey with a response rate of 20% may not be representative of the population.
• A survey with a high Cronbach’s Alpha value (e.g., 0.9) indicates high reliability.
• A survey with a low response rate (e.g., 10%) may indicate poor survey flow.
• A survey with a high validity value (e.g., 0.8) indicates that the survey is accurately measuring what it is intended to measure.
• A survey with a sample size of 1,000 respondents may not be necessary if the population is small.
• A survey with a high response rate (e.g., 90%) does not necessarily indicate high validity.