P-Hacking (Statistics / Research Methods)

Crash Course: P-Hacking (Statistics / Research Methods)

Crash Course: P-Hacking

Introduction Imagine you're a researcher trying to prove that eating pizza makes you smarter. You collect data, crunch the numbers, and – voilà! – you find a correlation between pizza consumption and IQ scores. Sounds legit, right? But what if I told you that correlation doesn't necessarily mean causation, and that your study might be a prime example of P-hacking?

The Core Idea P-hacking is the practice of manipulating data or statistical analysis to produce a desired outcome, often by cherry-picking results or ignoring inconvenient facts. It's like trying to find a needle in a haystack, but instead of looking for the needle, you're looking for a way to make the haystack look like it's got a needle.

Key Facts & Figures

• The term "P-hacking" was coined in 2011 by psychologist Brian Nosek, who wanted to describe the practice of manipulating statistical results to get a desired outcome.
• P-hacking is a form of research misconduct, and it's been linked to everything from fake news to fake science.
• In 2015, a study found that 71% of researchers admitted to engaging in P-hacking at some point in their careers.
• The most common forms of P-hacking include:
  • Data dredging: looking through a large dataset to find any correlation, no matter how weak.
  • HARKing: hypothesizing after results are known (i.e., looking at the data and then deciding what to test).
  • P-value hacking: manipulating the P-value to make a result look more significant.
• P-hacking can lead to false positives, which can have serious consequences in fields like medicine and finance.
• In 2018, a study found that 1 in 5 medical studies contained P-hacked results.
• P-hacking is not just a problem in science, but also in politics and journalism.
• The term "P-hacking" comes from the phrase "p-value hacking", which refers to the practice of manipulating the P-value to make a result look more significant.
• P-hacking can be done intentionally or unintentionally, but either way, it's a problem.
• In 2019, a study found that P-hacking was more common in fields with high stakes, such as medicine and finance.

Thought Bubble Imagine you're a researcher trying to prove that a new medication is effective. You collect data from a large group of patients, but when you look at the results, you find that the medication only works for people who are under 30. You decide to ignore the results for people over 30, because they're not relevant to your study. But what if I told you that ignoring those results is a form of P-hacking? You're manipulating the data to get a desired outcome, rather than looking at the whole picture.

Why This Matters

• P-hacking can lead to false positives, which can have serious consequences in fields like medicine and finance.
• P-hacking can undermine trust in science, which is essential for making informed decisions.
• P-hacking can lead to wasted resources, as researchers spend time and money on studies that are based on flawed data.
• P-hacking can be a form of intellectual dishonesty, which can damage a researcher's reputation and credibility.
• P-hacking can be a form of bias, which can lead to unfair conclusions and decisions.
• P-hacking can be a form of laziness, as researchers take the easy way out and ignore inconvenient facts.
• P-hacking can be a form of hubris, as researchers become so confident in their results that they ignore the possibility of error.

Crash Course Recap

• P-hacking is the practice of manipulating data or statistical analysis to produce a desired outcome.
• P-hacking is a form of research misconduct.
• P-hacking can lead to false positives.
• P-hacking can undermine trust in science.
• P-hacking can lead to wasted resources.
• P-hacking can be a form of intellectual dishonesty.
• P-hacking can be a form of bias.
• P-hacking can be a form of laziness.
• P-hacking can be a form of hubris.
• The term "P-hacking" was coined in 2011.
• 71% of researchers admit to engaging in P-hacking at some point in their careers.
• P-hacking is not just a problem in science, but also in politics and journalism.
• P-hacking can be done intentionally or unintentionally.
• P-hacking is more common in fields with high stakes.

Quiz Yourself

1. What is P-hacking? a) The practice of manipulating data or statistical analysis to produce a desired outcome. b) The practice of ignoring inconvenient facts. c) The practice of looking for correlations in a large dataset. d) The practice of hypothesizing after results are known.

Answer: a) The practice of manipulating data or statistical analysis to produce a desired outcome.

1. Who coined the term "P-hacking"? a) Brian Nosek b) John Ioannidis c) Andrew Gelman d) Nate Silver

Answer: a) Brian Nosek

1. What is the most common form of P-hacking? a) Data dredging b) HARKing c) P-value hacking d) All of the above

Answer: d) All of the above

1. What can P-hacking lead to? a) False positives b) False negatives c) Both false positives and false negatives d) Neither false positives nor false negatives

Answer: a) False positives

1. Why is P-hacking a problem? a) Because it's a form of intellectual dishonesty. b) Because it can lead to wasted resources. c) Because it can undermine trust in science. d) All of the above

Answer: d) All of the above