Hypothesis Testing > Unequal Sample Sizes
Problems with Unequal Sample Sizes
Unequally sized groups are common in research and may be the result of simple randomization, planned differences in group size or study dropouts. Unequal sample sizes can lead to:
- Unequal variances between samples, which affects the assumption of equal variances in tests like ANOVA. Having both unequal sample sizes and variances dramatically affects statistical power and Type I error rates (Rusticus & Lovato, 2014).
- A general loss of power. Equal-sized groups maximize statistical power.
- Issues with confounding variables.
Where, exactly this starts to matter isn’t clear. Keppel (1993) states that a rule of thumb for a magic cut-off point doesn’t seem to exist. That said, you don’t need equally sided groups to calculate accurate statistics, and most software will adjust for differences.
Some tests are set up specifically to deal with the problem of unequal sample sizes and unequal variances:
- Dunnett’s T3 or Dunnett’s C can be used for pairwise comparisons. Use T3 for small samples, and C for larger samples.
- Games-Howell Pairwise Comparison Test: an extension of the Tukey-Kramer test to handle unequal variances. Although it has more power (narrower confidence intervals) than Dunnett’s tests, alpha inflation can be a problem.
- Tamhane’s T2: combines Sidak’s multiplicative inequality test with Welch’s approximate solution.
- Welch’s Test for Unequal Variances is a modified Student’s t-test. The modified degrees of freedom tends to increase the test power for samples with unequal variance.
For unequal sample sizes that have equal variance, the following parametric post hoc tests can be used. All are considered conservative (Shingala):
- Dunnet’s test,
- Fisher’s test,
- Gabriel’s test.
- Hochberg’s GT2,
- Sidak’s test,
- Scheffe’s test,
- Tukey-Kramer test.
Non parametric options for unequal sample sizes are:
- Dunn pairwise,
- Dunn control,
Hochberg, Y. Tamhane, Y. Multiple Comparison Procedures, John Wiley & Sons, 1987.
Keppel, G. (1993). Design and Analysis: A Researcher’s Handbook. Pearson.
Parra-Frutos, I. Comput Stat (2013) Testing homogeneity of variances with unequal sample sizes. 28: 1269. doi:10.1007/s00180-012-0353-x.
Rusticus, S. & Lovato, C. (2014). Impact of Sample Size and variability on the Power and Type I Error Rates of Equivalence Tests: A Simulation Study. Practical Assessment, Research & Evaluation. Vol. 19, No. 11. August.
Shingala, C. et. al. / International Journal of New Technologies in Science and Engineering
Vol. 2, Issue 5,Nov 2015, ISSN 2349-0780
CITE THIS AS:
Stephanie Glen. "Unequal Sample Sizes" From StatisticsHowTo.com: Elementary Statistics for the rest of us! https://www.statisticshowto.com/unequal-sample-sizes/
Need help with a homework or test question? With Chegg Study, you can get step-by-step solutions to your questions from an expert in the field. Your first 30 minutes with a Chegg tutor is free!
Comments? Need to post a correction? Please Contact Us.
Can you do at test with unequal sample sizes? ›
The short answer: Yes, you can perform a t-test when the sample sizes are not equal. Equal sample sizes is not one of the assumptions made in a t-test. The real issues arise when the two samples do not have equal variances, which is one of the assumptions made in a t-test.How does unequal sample size affect power? ›
The power reduces as the group sizes become more and more unequal. So we get maximum power for a given Total sample size when the groups are of equal size.Is it better statistically to have unequal sample sizes or equal sample sizes? ›
It can be shown that the greater the differences in sample sizes between the groups, the lower the statistical power of an ANOVA. This is why researchers typically want equal sample sizes so that they have higher power and thus a greater probability of detecting true differences.How does unequal sample size affect ANOVA? ›
The main practical issue in one-way ANOVA is that unequal sample sizes affect the robustness of the equal variance assumption. ANOVA is considered robust to moderate departures from this assumption. But that's not true when the sample sizes are very different.How does different sample size affect results? ›
Too small a sample may prevent the findings from being extrapolated, whereas too large a sample may amplify the detection of differences, emphasizing statistical differences that are not clinically relevant.How do you compare two samples with different sizes? ›
Use a permuation test.
Randomly shuffle the values between the two groups, maintaining the original sample size. What fraction of those shuffled data sets have a difference between means as large (or larger) than observed. That is the P value.
The sample size for a study needs to be estimated at the time the study is proposed; too large a sample is unnecessary and unethical, and too small a sample is unscientific and also unethical. The necessary sample size can be calculated, using statistical software, based on certain assumptions.Why does sample size affect accuracy? ›
On any hypothesis, scientific research is built upon determining the mean values of a given dataset. The larger the sample size, the more accurate the average values will be. Larger sample sizes also help researchers identify outliers in data and provide smaller margins of error.Do sample sizes have to be equal? ›
You don't need equal-sized groups to compute accurate statistics. If the sample size imbalance is due to drop-outs rather than due to design, simple randomisation or technical glitches, this is something to take into account when interpreting the results. Whatever you do, don't throw away data.How know if two samples have unequal variance? ›
Two-sample T-Test with unequal variance can be applied when (1) the samples are normally distributed, (2) the standard deviation of both populations are unknown and assume to be unequal, and the (3) sample is sufficiently large (over 30).
Which sample size should give the most accurate results? ›
A good maximum sample size is usually around 10% of the population, as long as this does not exceed 1000. For example, in a population of 5000, 10% would be 500. In a population of 200,000, 10% would be 20,000. This exceeds 1000, so in this case the maximum would be 1000.Why does sample size affect variability? ›
However, the variability in the sample means will depend on the size of the samples, since larger samples are more likely to give estimated means that are closer to the true mean of the population.What is the problem with unequal sample size? ›
Unequal sample sizes can lead to: Unequal variances between samples, which affects the assumption of equal variances in tests like ANOVA. Having both unequal sample sizes and variances dramatically affects statistical power and Type I error rates (Rusticus & Lovato, 2014). A general loss of power.Can I do a two-way ANOVA with unequal sample sizes? ›
Yes it will work very well. ANOVA works with the mean and thus, does not bother whether samples are equal or not.How does sample size affect bias and variance? ›
The reason is that increasing sample size will not only reduce bias, but will eliminate the variance effects of sampling error. This low cost bias adjustment method in this study does not purge the effects that sampling error has on the variance in the same way that increasing the sample size does.What happens when you change sample size? ›
In other words, as the sample size increases, the variability of sampling distribution decreases. Also, as the sample size increases the shape of the sampling distribution becomes more similar to a normal distribution regardless of the shape of the population.Why is a larger sample size more accurate? ›
Larger samples more closely approximate the population. Because the primary goal of inferential statistics is to generalize from a sample to a population, it is less of an inference if the sample size is large.What is the best statistical test for comparing means between groups of unequal size? ›
anova - Best statistical test to compare two groups when they have different distributions - Cross Validated.What statistical test would you use to compare two groups of different sizes? ›
What Is a T-Test? A t-test is an inferential statistic used to determine if there is a significant difference between the means of two groups and how they are related.What test do you use to compare two samples? ›
What is the two-sample t-test? The two-sample t-test (also known as the independent samples t-test) is a method used to test whether the unknown population means of two groups are equal or not.
Does sample size affect reliability or validity? ›
Appropriate sample sizes are critical for reliable, reproducible, and valid results. Evidence generated from small sample sizes is especially prone to error, both false negatives (type II errors) due to inadequate power and false positives (type I errors) due to biased samples.Does sample size affect precision or accuracy? ›
If you increase your sample size you increase the precision of your estimates, which means that, for any given estimate / size of effect, the greater the sample size the more “statistically significant” the result will be.What is most affected by sample size? ›
Sample size refers to the number of participants or observations included in a study. This number is usually represented by n. The size of a sample influences two statistical properties: 1) the precision of our estimates and 2) the power of the study to draw conclusions.Can I use ANOVA for unbalanced data? ›
However, if you do have to perform an experiment using an unbalanced design, you have three choices: 1. Proceed with an ANOVA anyway. If the sample sizes across treatment combinations are not equal, but the assumption of equal variances is met, you can still proceed to perform an ANOVA anyway.What is Welch's test for unequal variances? ›
Welch's t-test also known as unequal variances t-test is used when you want to test whether the means of two population are equal. This test is generally applied when the there is a difference between the variations of two populations and also when their sample sizes are unequal.What does unequal variance mean? ›
The conservative choice is to use the "Unequal Variances" column, meaning that the data sets are not pooled. This doesn't require you to make assumptions that you can't really be sure of, and it almost never makes much of a change in your results.Should I assume equal or UNequal variance? ›
The Two-Sample assuming Equal Variances test is used when you know (either through the question or you have analyzed the variance in the data) that the variances are the same. The Two-Sample assuming UNequal Variances test is used when either: You know the variances are not the same.What does t-test two-sample assuming UNequal variances mean? ›
This tool executes a two-sample student's t-Test on data sets from two independent populations with unequal variances. This test can be either two-tailed or one-tailed contingent upon if we are testing that the two population means are different or if one is greater than the other.How do you determine if two populations variances are significantly different? ›
If the p-value is less than your significance level (e.g., 0.05), you can reject the null hypothesis. The difference between the two variances is statistically significant. This condition indicates that your sample provides strong enough evidence to conclude that the variability in the two populations are different.What is the best sample size for quantitative research? ›
Summary: 40 participants is an appropriate number for most quantitative studies, but there are cases where you can recruit fewer users.
Does sample size really matter? ›
A larger sample size should hypothetically lead to more accurate or representative results, but when it comes to surveying large populations, bigger isn't always better. In fact, trying to collect results from a larger sample size can add costs – without significantly improving your results.What reduces sampling variability? ›
As discussed earlier, use of larger random samples decreases the sample-to-sample variability and increases our confidence that the sample estimates are closer to the population parameters.How does sample size affect population variance? ›
That is, the variance of the sampling distribution of the mean is the population variance divided by N, the sample size (the number of scores used to compute a mean). Thus, the larger the sample size, the smaller the variance of the sampling distribution of the mean.How does sample size impact standard deviation? ›
Thus as the sample size increases, the standard deviation of the means decreases; and as the sample size decreases, the standard deviation of the sample means increases.What happens if the sampling is wrong? ›
A sampling error is a statistical error that occurs when an analyst does not select a sample that represents the entire population of data. As a result, the results found in the sample do not represent the results that would be obtained from the entire population.What are at least two problems associated with a sample size that is too small? ›
This is a real problem because small sample size is associated with: low statistical power. inflated false discovery rate. inflated effect size estimation.Can you do correlation with unequal sample sizes? ›
If you have two variables with different sizes, they are not paired, and it is not possible to calculate the correlation coefficient of both variables.Can you use one-way ANOVA if data is not normally distributed? ›
As regards the normality of group data, the one-way ANOVA can tolerate data that is non-normal (skewed or kurtotic distributions) with only a small effect on the Type I error rate.Does sample size need to be equal for ANOVA? ›
If you conduct an ANOVA test, you should always try to keep the same sample sizes for each factor level. A general rule of thumb for equal variances is to compare the smallest and largest sample standard deviations. This is much like the rule of thumb for equal variances for the test for independent means.What are the factors affecting sample size? ›
The factors affecting sample sizes are study design, method of sampling, and outcome measures – effect size, standard deviation, study power, and significance level.
How does sampling bias affect results? ›
It affects the internal validity of an analysis by leading to inaccurate estimation of relationships between variables. It also can affect the external validity of an analysis because the results from a biased sample may not generalize to the population.What biases could affect sampling data? ›
Some common types of sampling bias include self-selection bias, nonresponse bias, undercoverage bias, survivorship bias, pre-screening or advertising bias, and healthy user bias.Can Mann Whitney U test be used on unequal sample sizes? ›
Yes, the Mann-Whitney test works fine with unequal sample sizes.Can you do a paired t-test with unequal variances? ›
Two-sample T-Test with unequal variance can be applied when (1) the samples are normally distributed, (2) the standard deviation of both populations are unknown and assume to be unequal, and the (3) sample is sufficiently large (over 30).Do you need sample size for t-test? ›
No. There is no minimum sample size required to perform a t-test. In fact, the first t-test ever performed only used a sample size of four. However, if the assumptions of a t-test are not met then the results could be unreliable.Does a two sample t test have equal or unequal variance? ›
Two-sample t-test assumptions
Data in each group must be obtained via a random sample from the population. Data in each group are normally distributed. Data values are continuous. The variances for the two independent groups are equal.
The Mann–Whitney U test is applied to independent samples. The Wilcoxon signed-rank test is applied to matched or dependent samples.When should I use Wilcoxon-Mann-Whitney test? ›
The Mann-Whitney (or Wilcoxon-Mann-Whitney) test is sometimes used for comparing the efficacy of two treatments in clinical trials. It is often presented as an alternative to a t test when the data are not normally distributed.Does Mann Whitney need equal sample sizes? ›
10.4 Statistical analysis for comparison of probes
The Mann-Whitney (Wilcoxon rank-sum test) should be used for comparison between cohorts with sample sizes greater than 30, significantly different sample sizes, or if a normal distribution of samples cannot be assumed.
Use the unequal variance t test, also called the Welch t test. It assumes that both groups of data are sampled from Gaussian populations, but does not assume those two populations have the same standard deviation.
What if variances are not equal? ›
Unequal variances (heteroscedasticity) can affect the Type I error rate and lead to false positives. If you are comparing two or more sample means, as in the 2-Sample t-test and ANOVA, a significantly different variance could overshadow the differences between means and lead to incorrect conclusions.What is the hypothesis test for unequal variances? ›
In statistics, Welch's t-test, or unequal variances t-test, is a two-sample location test which is used to test the (null) hypothesis that two populations have equal means.Do sample sizes need to be equal? ›
You don't need equal-sized groups to compute accurate statistics. If the sample size imbalance is due to drop-outs rather than due to design, simple randomisation or technical glitches, this is something to take into account when interpreting the results. Whatever you do, don't throw away data.What if the sample size is not enough? ›
A study with an insufficient sample size may not have sufficient statistical power to detect meaningful effects and may produce unreliable answers to important research questions. On the other hand, a study with an excessive sample size wastes resources and may unnecessarily expose study participants to potential harm.What is the minimum sample size for statistical analysis? ›
The minimum sample size is 100
Most statisticians agree that the minimum sample size to get any kind of meaningful result is 100. If your population is less than 100 then you really need to survey all of them.
Unequal sample sizes can lead to: Unequal variances between samples, which affects the assumption of equal variances in tests like ANOVA. Having both unequal sample sizes and variances dramatically affects statistical power and Type I error rates (Rusticus & Lovato, 2014). A general loss of power.What does t-test two sample assuming unequal variances mean? ›
This tool executes a two-sample student's t-Test on data sets from two independent populations with unequal variances. This test can be either two-tailed or one-tailed contingent upon if we are testing that the two population means are different or if one is greater than the other.How do you find the degrees of freedom for unequal sample sizes? ›
A simplified way for calculating it is df= min(n1-1,n2-1).