Cross-Domain Uncertainty Evaluation in Predictive Analytics: Credit Default and Customer Churn
Article Sidebar
Main Article Content
Abstract
This study develops and evaluates a cross-domain, probability-based framework for evaluating uncertainty in predictive analytics. Although predictive models widely support decision-making, many models are evaluated mainly using accuracy-based indicators that do not show whether predicted probabilities are reliable, calibrated, or suitable for decision support. The proposed framework integrates probabilistic classification, Bayesian band-level estimation, calibration diagnostics, bootstrap validation, scoring rules, and probability-based decision bands. Two public benchmark datasets were used: a credit-card default dataset with 30,000 observations as the main financial-risk case and a customer-churn dataset with 7,043 observations as a secondary marketing-analytics validation case. Logistic regression, random forest, and gradient boosting models were compared using conventional metrics and uncertainty-aware indicators, including the area under the receiver operating characteristic curve (ROC-AUC), accuracy, precision, recall, F1-score, Brier score, log loss, expected calibration error, maximum calibration error, bootstrap intervals, calibration curves, and Bayesian credible intervals for observed event rates within probability bands. The results show that gradient boosting provided the strongest probability reliability in both empirical cases, with a Brier score of 0.1380 and expected calibration error of 0.0179 in credit default prediction and a Brier score of 0.1407 and expected calibration error of 0.0346 in customer churn prediction. Decision-band analysis showed that borderline cases had observed event rates near 54.33% for credit default and 52.43% for customer churn, confirming that uncertainty bands can separate confident predictions from review-needed cases. The study advances trustworthy predictive analytics by showing that model evaluation should combine predictive accuracy with probability reliability, calibration quality, and uncertainty communication.
Article Details
Issue
Section
This work is licensed under a Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License.
How to Cite
References
1. Angelopoulos, A. N., & Bates, S. (2021). A gentle introduction to conformal prediction and distribution-free uncertainty quantification. arXiv. DOI: 10.48550/arXiv.2107.07511.
2. Ashukha, A., Lyzhov, A., Molchanov, D., & Vetrov, D. (2020). Pitfalls of in-domain uncertainty estimation and ensembling in deep learning. International Conference on Learning Representations.
3. Blei, D. M., Kucukelbir, A., & McAuliffe, J. D. (2017). Variational inference: A review for statisticians. Journal of the American Statistical Association, 112(518), 859-877. DOI: 10.1080/01621459.2017.1285773. DOI: https://doi.org/10.1080/01621459.2017.1285773
4. Carpenter, B., Gelman, A., Hoffman, M. D., Lee, D., Goodrich, B., Betancourt, M., Brubaker, M., Guo, J., Li, P., & Riddell, A. (2017). Stan: A probabilistic programming language. Journal of Statistical Software, 76(1), 1-32. DOI: 10.18637/jss.v076.i01. DOI: https://doi.org/10.18637/jss.v076.i01
5. Collins, G. S., Reitsma, J. B., Altman, D. G., & Moons, K. G. M. (2015). Transparent reporting of a multivariable prediction model for individual prognosis or diagnosis: The TRIPOD statement. Annals of Internal Medicine, 162(1), 55-63. DOI: 10.7326/M14-0697. DOI: https://doi.org/10.7326/M14-0697
6. Gal, Y., & Ghahramani, Z. (2016). Dropout as a Bayesian approximation: Representing model uncertainty in deep learning. Proceedings of the 33rd International Conference on Machine Learning, 48, 1050-1059.
7. Gelman, A., Carlin, J. B., Stern, H. S., Dunson, D. B., Vehtari, A., & Rubin, D. B. (2013). Bayesian data analysis (3rd ed.). CRC Press. DOI: 10.1201/b16018. DOI: https://doi.org/10.1201/b16018
8. Guo, C., Pleiss, G., Sun, Y., & Weinberger, K. Q. (2017). On calibration of modern neural networks. Proceedings of the 34th International Conference on Machine Learning, 70, 1321-1330.
9. Hullermeier, E., & Waegeman, W. (2021). Aleatoric and epistemic uncertainty in machine learning: An introduction to concepts and methods. Machine Learning, 110, 457-506. DOI: 10.1007/s10994-021-05946-3. DOI: https://doi.org/10.1007/s10994-021-05946-3
10. Kendall, A., & Gal, Y. (2017). What uncertainties do we need in Bayesian deep learning for computer vision? Advances in Neural Information Processing Systems, 30, 5574-5584.
11. Kruschke, J. K. (2014). Doing Bayesian data analysis: A tutorial with R, JAGS, and Stan (2nd ed.). Academic Press. DOI: https://doi.org/10.1016/B978-0-12-405888-0.00008-8
12. Kuhn, M., & Johnson, K. (2013). Applied predictive modeling. Springer. DOI: 10.1007/978-1-4614-6849-3. DOI: https://doi.org/10.1007/978-1-4614-6849-3
13. Kuleshov, V., Fenner, N., & Ermon, S. (2018). Accurate uncertainties for deep learning using calibrated regression. Proceedings of the 35th International Conference on Machine Learning, 80, 2796-2804.
14. Kumar, A., Liang, P. S., & Ma, T. (2018). Verified uncertainty calibration. Advances in Neural Information Processing Systems, 31, 3792-3803.
15. Lakshminarayanan, B., Pritzel, A., & Blundell, C. (2017). Simple and scalable predictive uncertainty estimation using deep ensembles. Advances in Neural Information Processing Systems, 30, 6402-6413.
16. Leibig, C., Allken, V., Ayhan, M. S., Berens, P., & Wahl, S. (2017). Leveraging uncertainty information from deep neural networks for disease detection. Scientific Reports, 7, 17816. DOI: 10.1038/s41598-017-17876-z. DOI: https://doi.org/10.1038/s41598-017-17876-z
17. Mehrtash, A., Wells, W. M., Tempany, C. M., Abolmaesumi, P., & Kapur, T. (2020). Confidence calibration and predictive uncertainty estimation for deep medical image segmentation. IEEE Transactions on Medical Imaging, 39(12), 3868-3878. DOI: 10.1109/TMI.2020.3006437. DOI: https://doi.org/10.1109/TMI.2020.3006437
18. Moons, K. G. M., Altman, D. G., Reitsma, J. B., Ioannidis, J. P. A., Macaskill, P., Steyerberg, E. W., Vickers, A. J., Ransohoff, D. F., & Collins, G. S. (2015). Transparent reporting of a multivariable prediction model for individual prognosis or diagnosis: Explanation and elaboration. Annals of Internal Medicine, 162(1), W1-W73. DOI: 10.7326/M14-0698. DOI: https://doi.org/10.7326/M14-0698
19. Naeini, M. P., Cooper, G. F., & Hauskrecht, M. (2015). Obtaining well-calibrated probabilities using Bayesian binning. Proceedings of the 29th AAAI Conference, 2901-2907. DOI: 10.1609/aaai.v29i1.9602. DOI: https://doi.org/10.1609/aaai.v29i1.9602
20. Nixon, J., Dusenberry, M. W., Zhang, L., Jerfel, G., & Tran, D. (2019). Measuring calibration in deep learning. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, 38-41.
21. Ovadia, Y., Fertig, E., Ren, J., Nado, Z., Sculley, D., Nowozin, S., Dillon, J. V., Lakshminarayanan, B., & Snoek, J. (2019). Can you trust your model’s uncertainty? Evaluating predictive uncertainty under dataset shift. Advances in Neural Information Processing Systems, 32, 13991-14002.
22. Ribeiro, M. T., Singh, S., & Guestrin, C. (2016). Why should I trust you? Explaining the predictions of any classifier. Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, 1135-1144. DOI: 10.1145/2939672.2939778. DOI: https://doi.org/10.1145/2939672.2939778
23. Riley, R. D., Snell, K. I. E., Ensor, J., Burke, D. L., Harrell, F. E., Moons, K. G. M., & Collins, G. S. (2019). Minimum sample size for developing a multivariable prediction model: Part II - binary and time-to-event outcomes. Statistics in Medicine, 38(7), 1276-1296. DOI: 10.1002/sim.7992. DOI: https://doi.org/10.1002/sim.7992
24. Senge, R., Bosner, S., Dembczynski, K., Haasenritter, J., Hirsch, O., Donner-Banzhoff, N., & Hullermeier, E. (2014). Reliable classification: Learning classifiers that distinguish aleatoric and epistemic uncertainty. Information Sciences, 255, 16-29. DOI: 10.1016/j.ins.2013.07.030. DOI: https://doi.org/10.1016/j.ins.2013.07.030
25. Thulasidasan, S., Chennupati, G., Bilmes, J., Bhattacharya, T., & Michalak, S. (2019). On mixup training: Improved calibration and predictive uncertainty for deep neural networks. Advances in Neural Information Processing Systems, 32, 13888-13899. DOI: https://doi.org/10.2172/1525811
26. Vaicenavicius, J., Widmann, D., Andersson, C., Lindsten, F., Roll, J., & Schon, T. B. (2019). Evaluating model calibration in classification. Proceedings of Machine Learning Research, 89, 3459-3467.
27. Van Calster, B., McLernon, D. J., van Smeden, M., Wynants, L., & Steyerberg, E. W. (2019). Calibration: The Achilles heel of predictive analytics. BMC Medicine, 17, 230. DOI: 10.1186/s12916-019-1466-7. DOI: https://doi.org/10.1186/s12916-019-1466-7
28. Vehtari, A., Gelman, A., & Gabry, J. (2017). Practical Bayesian model evaluation using leave-one-out cross-validation and WAIC. Statistics and Computing, 27(5), 1413-1432. DOI: 10.1007/s11222-016-9696-4. DOI: https://doi.org/10.1007/s11222-016-9696-4
29. Wynants, L., Van Calster, B., Collins, G. S., Riley, R. D., Heinze, G., Schuit, E., Bonten, M. M. J., Damen, J. A. A. G., Debray, T. P. A., De Vos, M., Dhiman, P., Haller, M. C., Harhay, M. O., Henckaerts, L., Kreuzberger, N., Lohmann, A., Luijken, K., Ma, J., Martin, G. P., … van Smeden, M. (2020). Prediction models for diagnosis and prognosis of covid-19: Systematic review and critical appraisal. BMJ, 369, m1328. DOI: 10.1136/bmj.m1328. DOI: https://doi.org/10.1136/bmj.m1328
30. Yao, Y., Vehtari, A., Simpson, D., & Gelman, A. (2018). Using stacking to average Bayesian predictive distributions. Bayesian Analysis, 13(3), 917-1007. DOI: 10.1214/17-BA1091. DOI: https://doi.org/10.1214/17-BA1091