15.19
Statistical Significance
Once data is collected from both the experimental and the control groups, a statistical analysis is conducted to find out if there are meaningful diff…
After researchers collect empirical data from a predetermined sample size, they can follow up with statistical analyses to determine whether the differences they observed between groups or variables are meaningful or the result of chance alone.
For example, perhaps a researcher finds that undergraduate students who were asked to use gesture and emotional expression to act out a scene—the experimental group—remembered more of their lines than students who read them without using gesture and emotional expression—the control group.
Now she wants to know the likelihood that this difference occurred because the experimental manipulation affected participants’ memory for the lines, rather than because of random happenstance.
To accomplish this task, she needs to establish the p-value—the probability that the difference between the groups occurred by chance.
If she finds that the p-value is 0.05 or below—meaning there is a five percent or less possibility the result occurred by chance—then the difference between the groups is considered statistically significant by convention.
As a result, she can confidently accept the alternative or experimental hypothesis—that the experimental manipulation affected the results—and reject the null hypothesis—that the experimental manipulation had no effect.
In the end, if research findings are found to be statistically significant, the results are considered meaningful differences by the scientific community.
View the full transcript and gain access to JoVE Core videos
Q1: What does it mean when research results are statistically significant?
Statistical significance means the difference between groups is unlikely due to chance alone. In psychology, results are considered statistically significant when there is a 5 percent or less probability the difference occurred by chance. This allows researchers to confidently accept the experimental hypothesis and reject the null hypothesis, indicating the independent variable caused the observed effect.
Q2: How do researchers determine if observed differences are meaningful?
Researchers calculate the p-value, which represents the probability that differences between groups occurred by chance. If the p-value is 0.05 or below, the difference is considered statistically significant and meaningful. This threshold allows researchers to distinguish between real effects and random variation in their data.
Q3: Why can experiments establish cause and effect relationships?
Experiments establish cause and effect relationships because random selection, random assignment, and controlled designs create similar groups that differ only in the independent variable. When statistically significant differences emerge between groups, researchers can confidently attribute them to the experimental manipulation rather than confounding factors, enabling causal claims about the independent variable's effect.
Q4: What is the p-value and why is the 0.05 threshold important?
The p-value is the probability that observed differences between groups occurred by chance. The 0.05 threshold means researchers accept results as statistically significant only when there is a 5 percent or less likelihood of chance occurrence. This conventional standard protects against false conclusions and ensures findings are reliable enough for the scientific community to consider them meaningful.
Q5: How does statistical analysis connect experimental design to meaningful conclusions?
Statistical analysis evaluates whether observed differences between experimental and control groups are meaningful or random. By calculating p-values, researchers determine if their findings meet the 0.05 significance threshold. When results are statistically significant, the experimental design's controls ensure differences stem from the independent variable, allowing researchers to make confident causal statements about their findings.
Q6: What happens if a study's results are not statistically significant?
If results are not statistically significant, the p-value exceeds 0.05, meaning there is greater than a 5 percent probability the difference occurred by chance. Researchers cannot confidently reject the null hypothesis or accept the experimental hypothesis. The observed differences are considered unreliable, and researchers cannot claim the independent variable caused the effect.
Q7: How does sample size relate to determining statistical significance?
Researchers establish a predetermined sample size before collecting data to ensure adequate statistical power. A sufficiently large sample reduces the influence of random variation and increases the likelihood of detecting true effects if they exist. Proper sample sizing strengthens the ability to determine whether observed differences are statistically significant or merely due to chance.