Large-Scale Hypothesis Testing and Confidence Interval Estimation: An Effect-Size and Multiplicity-Aware Framework
Article Sidebar
Main Article Content
Abstract
This study develops and empirically evaluates an integrated framework for statistical hypothesis testing and confidence interval estimation in large-scale data analysis. Using the 2015 Behavioral Risk Factor Surveillance System (BRFSS) binary diabetes health-indicator file, the analysis examined 253,680 observations and 22 variables, including diabetes status, cardiometabolic risk indicators, lifestyle indicators, healthcare-access measures, and socioeconomic variables. The analysis used descriptive statistics, prevalence estimates, Welch mean-comparison tests, chi-square tests, confidence intervals, standardized effect sizes, odds ratios, risk differences, multiple-comparison correction, and adjusted logistic regression. Diabetes prevalence in the main file was 13.93% (95% CI, 13.80% to 14.07%). The results showed that the large sample size produced very small p-values for most comparisons; however, effect magnitudes varied substantially across variables. Body mass index (BMI) was higher among respondents with diabetes by 4.14 units, with a 95% confidence interval of 4.06 to 4.22 and a moderate standardized effect. High blood pressure showed a strong unadjusted association with diabetes (odds ratio 5.04; 95% CI, 4.91 to 5.17), while physical activity showed a lower odds pattern for diabetes. After adjustment, high blood pressure, high cholesterol, general health, age, and BMI remained important predictors. The study demonstrates that hypothesis testing is most reliable when p-values are interpreted together with confidence intervals, effect sizes, adjusted estimates, and multiplicity-aware decision rules. The framework provides a reproducible model to strengthen statistical interpretation in large empirical datasets.
Article Details
Issue
Section
This work is licensed under a Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License.
How to Cite
References
1. Amrhein, V., Greenland, S., & McShane, B. (2019). Scientists rise up against statistical significance. Nature, 567(7748), 305–307. https://doi.org/10.1038/d41586-019-00857-9 DOI: https://doi.org/10.1038/d41586-019-00857-9
2. Amrhein, V., Trafimow, D., & Greenland, S. (2019). Inferential statistics as descriptive statistics: There is no replication crisis if we do not expect replication. The American Statistician, 73(sup1), 262–270. https://doi.org/10.1080/00031305.2018.1543137 DOI: https://doi.org/10.1080/00031305.2018.1543137
3. Baker, M. (2016). 1,500 scientists lift the lid on reproducibility. Nature, 533(7604), 452–454. https://doi.org/10.1038/533452a DOI: https://doi.org/10.1038/533452a
4. Benjamin, D. J., Berger, J. O., Johannesson, M., Nosek, B. A., Wagenmakers, E. J., Berk, R., Bollen, K. A., Brembs, B., Brown, L., Camerer, C., Cesarini, D., Chambers, C. D., Clyde, M., Cook, T. D., De Boeck, P., Dienes, Z., Dreber, A., Easwaran, K., Efferson, C., ... Johnson, V. E. (2018). Redefine statistical significance. Nature Human Behaviour, 2(1), 6–10. https://doi.org/10.1038/s41562-017-0189-z DOI: https://doi.org/10.1038/s41562-017-0189-z
5. Calin-Jageman, R. J., & Cumming, G. (2019). Estimation for better inference in neuroscience. eNeuro, 6(4), ENEURO.0205-19.2019. https://doi.org/10.1523/ENEURO.0205-19.2019 DOI: https://doi.org/10.1523/ENEURO.0205-19.2019
6. Centers for Disease Control and Prevention. (2015). Behavioral Risk Factor Surveillance System survey data. U.S. Department of Health and Human Services.
7. Colquhoun, D. (2014). An investigation of the false discovery rate and the misinterpretation of p-values. Royal Society Open Science, 1(3), 140216. https://doi.org/10.1098/rsos.140216 DOI: https://doi.org/10.1098/rsos.140216
8. Cumming, G. (2014). The new statistics: Why and how. Psychological Science, 25(1), 7–29. https://doi.org/10.1177/0956797613504966 DOI: https://doi.org/10.1177/0956797613504966
9. Fan, J., Han, F., & Liu, H. (2014). Challenges of big data analysis. National Science Review, 1(2), 293–314. https://doi.org/10.1093/nsr/nwt032 DOI: https://doi.org/10.1093/nsr/nwt032
10. Funder, D. C., & Ozer, D. J. (2019). Evaluating effect size in psychological research: Sense and nonsense. Advances in Methods and Practices in Psychological Science, 2(2), 156–168. https://doi.org/10.1177/2515245919847202 DOI: https://doi.org/10.1177/2515245919847202
11. Gelman, A., & Carlin, J. (2014). Beyond power calculations: Assessing Type S and Type M errors. Perspectives on Psychological Science, 9(6), 641–651. https://doi.org/10.1177/1745691614551642 DOI: https://doi.org/10.1177/1745691614551642
12. Goeman, J. J., & Solari, A. (2014). Multiple hypothesis testing in genomics. Statistics in Medicine, 33(11), 1946–1978. https://doi.org/10.1002/sim.6082 DOI: https://doi.org/10.1002/sim.6082
13. Greenland, S., Senn, S. J., Rothman, K. J., Carlin, J. B., Poole, C., Goodman, S. N., & Altman, D. G. (2016). Statistical tests, p values, confidence intervals, and power: A guide to misinterpretations. European Journal of Epidemiology, 31(4), 337–350. https://doi.org/10.1007/s10654-016-0149-3 DOI: https://doi.org/10.1007/s10654-016-0149-3
14. Halsey, L. G., Curran-Everett, D., Vowler, S. L., & Drummond, G. B. (2015). The fickle p-value generates irreproducible results. Nature Methods, 12(3), 179–185. https://doi.org/10.1038/nmeth.3288 DOI: https://doi.org/10.1038/nmeth.3288
15. Ho, J., Tumkaya, T., Aryal, S., Choi, H., & Claridge-Chang, A. (2019). Moving beyond p values: Data analysis with estimation graphics. Nature Methods, 16(7), 565–566. https://doi.org/10.1038/s41592-019-0470-3 DOI: https://doi.org/10.1038/s41592-019-0470-3
16. Hurlbert, S. H., Levine, R. A., & Utts, J. (2019). Coup de grâce for a tough old bull: Statistically significant expires. The American Statistician, 73(sup1), 352–357. https://doi.org/10.1080/00031305.2018.1543616 DOI: https://doi.org/10.1080/00031305.2018.1543616
17. Ioannidis, J. P. A. (2018). Proposal to lower p-value thresholds to .005. JAMA, 319(14), 1429–1430. https://doi.org/10.1001/jama.2018.1536 DOI: https://doi.org/10.1001/jama.2018.1536
18. Lakens, D., Scheel, A. M., & Isager, P. M. (2018). Equivalence testing for psychological research: A tutorial. Advances in Methods and Practices in Psychological Science, 1(2), 259–269. https://doi.org/10.1177/2515245918770963 DOI: https://doi.org/10.1177/2515245918770963
19. Lakens, D. (2022). Sample size justification. Collabra: Psychology, 8(1), 33267. https://doi.org/10.1525/collabra.33267 DOI: https://doi.org/10.1525/collabra.33267
20. Leek, J. T., & Peng, R. D. (2015). Statistics: P values are just the tip of the iceberg. Nature, 520(7549), 612. https://doi.org/10.1038/520612a DOI: https://doi.org/10.1038/520612a
21. Makin, T. R., & Orban de Xivry, J. J. (2019). Ten common statistical mistakes to watch out for when writing or reviewing a manuscript. eLife, 8, e48175. https://doi.org/10.7554/eLife.48175 DOI: https://doi.org/10.7554/eLife.48175
22. Mansournia, M. A., Collins, G. S., Nielsen, R. O., Nazemipour, M., Jewell, N. P., Altman, D. G., & Campbell, M. J. (2021). Checklist for Statistical Assessment of Medical Papers: The CHAMP statement. British Journal of Sports Medicine, 55(18), 1002–1003. https://doi.org/10.1136/bjsports-2020-103651 DOI: https://doi.org/10.1136/bjsports-2020-103651
23. McShane, B. B., Gal, D., Gelman, A., Robert, C., & Tackett, J. L. (2019). Abandon statistical significance. The American Statistician, 73(sup1), 235–245. https://doi.org/10.1080/00031305.2018.1527253 DOI: https://doi.org/10.1080/00031305.2018.1527253
24. Meng, X. L. (2018). Statistical paradises and paradoxes in big data: Law of large populations, big data paradox, and the 2016 US presidential election. The Annals of Applied Statistics, 12(2), 685–726. https://doi.org/10.1214/18-AOAS1161SF DOI: https://doi.org/10.1214/18-AOAS1161SF
25. Morey, R. D., Hoekstra, R., Rouder, J. N., Lee, M. D., & Wagenmakers, E. J. (2016). The fallacy of placing confidence in confidence intervals. Psychonomic Bulletin & Review, 23(1), 103–123. https://doi.org/10.3758/s13423-015-0947-8 DOI: https://doi.org/10.3758/s13423-015-0947-8
26. Munafò, M. R., Nosek, B. A., Bishop, D. V. M., Button, K. S., Chambers, C. D., du Sert, N. P., Simonsohn, U., Wagenmakers, E. J., Ware, J. J., & Ioannidis, J. P. A. (2017). A manifesto for reproducible science. Nature Human Behaviour, 1, 0021. https://doi.org/10.1038/s41562-016-0021 DOI: https://doi.org/10.1038/s41562-016-0021
27. Nuzzo, R. (2014). Scientific method: Statistical errors. Nature, 506(7487), 150–152. https://doi.org/10.1038/506150a DOI: https://doi.org/10.1038/506150a
28. Rafi, Z., & Greenland, S. (2020). Semantic and cognitive tools to aid statistical science: Replace confidence and significance by compatibility and surprise. BMC Medical Research Methodology, 20, 244. https://doi.org/10.1186/s12874-020-01105-9 DOI: https://doi.org/10.1186/s12874-020-01105-9
29. Schäfer, T., & Schwarz, M. A. (2019). The meaningfulness of effect sizes in psychological research: Differences between sub-disciplines and the impact of potential biases. Frontiers in Psychology, 10, 813. https://doi.org/10.3389/fpsyg.2019.00813 DOI: https://doi.org/10.3389/fpsyg.2019.00813
30. Szucs, D., & Ioannidis, J. P. A. (2017). Empirical assessment of published effect sizes and power in the recent cognitive neuroscience and psychology literature. PLOS Biology, 15(3), e2000797. https://doi.org/10.1371/journal.pbio.2000797 DOI: https://doi.org/10.1371/journal.pbio.2000797
31. Teboul, A. (2021). Diabetes Health Indicators Dataset. Kaggle. https://www.kaggle.com/datasets/alexteboul/diabetes-health-indicators-dataset
32. Trafimow, D., & Earp, B. D. (2017). Null hypothesis significance testing and Type I error: The domain problem. Frontiers in Psychology, 8, 621. https://doi.org/10.3389/fpsyg.2017.00621 DOI: https://doi.org/10.1016/j.newideapsych.2017.01.002
33. Wasserstein, R. L., & Lazar, N. A. (2016). The ASA statement on p-values: Context, process, and purpose. The American Statistician, 70(2), 129–133. https://doi.org/10.1080/00031305.2016.1154108 DOI: https://doi.org/10.1080/00031305.2016.1154108
34. Wasserstein, R. L., Schirm, A. L., & Lazar, N. A. (2019). Moving to a world beyond p < .05. The American Statistician, 73(sup1), 1–19. https://doi.org/10.1080/00031305.2019.1583913 DOI: https://doi.org/10.1080/00031305.2019.1583913