Citation :

Khaoula Echabbi, Mohammed Douimi, El Moukhtar Zemmouri, salsabil Hamdi, "A Hybrid 3D CNN and Clinical Data Fusion Model for COVID-19 Mortality Prediction," International Journal of Engineering Trends and Technology (IJETT), vol. 74, no. 8, pp. 182-196, 2026. Crossref, https://doi.org/10.14445/22315381/IJETT-V74I8P112

Abstract

The COVID-19 pandemic revealed acute flaws in hospitals' decision-making processes, exposing substantial deficiencies in how critically ill patients are managed or in one dimensionalizing complex multidimensional data in a timely manner. Many clinical scoring systems such as the Sequential Organ Failure Assessment (SOFA) and the Acute Physiology and Chronic Health Evaluation II (APACHE-II) are still very widely used based upon their implementation in routine clinical practice, however, both methods are limited by their reliance on a small number of predetermined clinical characteristics as well as not allowing for extraction of high dimensional, data from multiple complex data sources (e.g., volumetric CT). Significant advancements in deep learning techniques have provided significant advancements in terms of both the potential applications as well as how accurately and automatically images can be interpreted. However, most of these methods are limited by only using a single image modality (i.e., either imaging data or clinical data) and therefore provide limited performance since characteristics of either type of data represent only part of the overall information available from the combination of both types of data. To address these limitations, we propose a multimodal deep learning framework for assessing the probability of death in COVID-19 patients that utilizes volumetric CT imaging in conjunction with structured clinical data and integrates a 3D convolutional neural network for CT image feature representation and a dense neural network for clinical representation of clinical data. In order to ensure consistent and reliable data input for our models, we have developed a dedicated preprocessing pipeline for our data that includes lung segmentation and Hounsfield unit normalisation. Results of our experiments completed on Moroccan COVID-19 patient data show that our multimodal approach outperformed the unimodal approach, suggesting that data fusion with respect to predicting risk in a clinical context provides significant benefit. In addition, our proposed framework will be generalisable and may be applicable to many other areas of medicine requiring the integration of multimodal datasets.

Keywords

Artificial intelligence, Computed tomography, Deep learning, Medical imaging, Multimodal learning.

References

[1] I Ketut Parindra, Menap Menap, and Lalu Abdul Khalik, “Optimization of Hospital Health Services: Insights from Recent Literature,” Bioscientist: Scientific Journal of Biology, vol. 12, no. 2, pp. 1636-1655, 2024.
[CrossRef] [Google Scholar] [Publisher Link]

[2] Tao Ai et al., “Correlation of Chest CT and RT-PCR Testing for Coronavirus Disease 2019 (COVID-19) in China: A Report of 1014 Cases,” Radiology, vol. 296, no. 2, pp. 32-40, 2020.
[CrossRef] [Google Scholar] [Publisher Link]

[3] Laure Wynants et al., “Prediction Models for Diagnosis and Prognosis of COVID-19 Infection: Systematic Review and Critical Appraisal,” BMJ, vol. 369, pp. 1-22, 2020.
[CrossRef] [Google Scholar] [Publisher Link]

[4] O.S. Albahri et al., “Systematic Review of Artificial Intelligence Techniques in the Detection and Classification of COVID-19 Medical Images in Terms of Evaluation and Benchmarking: Taxonomy Analysis, Challenges, Future Solutions and Methodological Aspects,” Journal of Infection and Public Health, vol. 13, no. 10, pp. 1381-1396, 2020. [CrossRef] [Google Scholar] [Publisher Link]

[5] Viswajeetsinh Jadeja et al., “Convolutional Neural Networks: An Extensive Review of Architectures and their Applications,” 2023 6^th International Conference on Contemporary Computing and Informatics, Gautam Buddha Nagar, India, pp. 460-467, 2023.
[CrossRef] [Google Scholar] [Publisher Link] 

[6] Marta Villegas et al., “Predicting the Evolution of COVID-19 Mortality Risk: A Recurrent Neural Network Approach,” Computer Methods and Programs in Biomedicine Update, vol. 3, pp. 1-9, 2022.
[CrossRef] [Google Scholar] [Publisher Link]

[7] Feng Shi et al., “Review of Artificial Intelligence Techniques in Imaging Data Acquisition, Segmentation, and Diagnosis for COVID-19,” IEEE Reviews in Biomedical Engineering, vol. 14, pp. 4-15, 2021.
[CrossRef] [Google Scholar] [Publisher Link] 

[8] Yao-Hung Hubert Tsai et al., “Multimodal Transformer for Unaligned Multimodal Language Sequences,” Proceedings of the 57^th Annual Meeting of the Association for Computational Linguistics, Association for Computational Linguistics, Florence, Italy, pp. 6558-6569, 2019.
[CrossRef] [Google Scholar] [Publisher Link]

[9] Mahbubunnabi Tamal et al., “Artificial Intelligence (AI) Based Prediction of Mortality, ICU Admission and Ventilation Support Requirement for COVID-19 Patients Using 122 Clinical and Demographic Parameters,” medRxiv, pp. 1-17, 2024.
[CrossRef] [Google Scholar] [Publisher Link]

[10] Marwah Ahmed Halwani, and Manal Ahmed Halwani, “Prediction of COVID-19 Hospitalization and Mortality Using Artificial Intelligence,” Healthcare, vol. 12, no. 17, pp. 1-14, 2024.
[CrossRef] [Google Scholar] [Publisher Link]

[11] Saeed Shakibfar et al., “Artificial Intelligence-Driven Prediction of COVID-19-Related Hospitalization and Death: A Systematic Review,” Frontiers in Public Health, vol. 11, pp. 1-15, 2023.
[CrossRef] [Google Scholar] [Publisher Link]

[12] Kagan Tur et al., “Multi-Modal Machine Learning Approach for COVID-19 Detection using Clinical Biomarkers and Chest X-Ray Images,” Diagnostics, vol. 14, no. 24, pp. 1-13, 2024.
[CrossRef] [Google Scholar] [Publisher Link]

[13] Albatoul S. Althenayan et al., “COVID-19 Hierarchical Classification Using a Deep Learning Multi-Modal,” Sensors, vol. 24, no. 8, pp. 1-30, 2024.
[CrossRef] [Google Scholar] [Publisher Link]

[14] Shih-Cheng Huang et al., “Fusion of Medical Imaging and Electronic Health Records using Deep Learning: A Systematic Review and Implementation Guidelines,” npj Digital Medicine, vol. 3, no. 1, pp. 1-9, 2020.
[CrossRef] [Google Scholar] [Publisher Link]

[15] Akshaya Karthikeyan et al., “Machine Learning based Clinical Decision Support System for Early COVID-19 Mortality Prediction,” Frontiers in Public Health, vol. 9, pp. 1-13, 2021.
[CrossRef] [Google Scholar] [Publisher Link]

[16] Tami D. DenOtter, and Johanna Schubert, Hounsfield Unit, StatPearls, 2023.
[Google Scholar]

[17] Joaquín Pérez-Ortega, Nelva Nely Almanza-Ortega, and David Romero, “Balancing Effort and Benefit of K-Means Clustering Algorithms in Big Data Realms,” PLOS One, vol. 13, no. 9, pp. 1-19, 2018.
[CrossRef] [Google Scholar] [Publisher Link]

[18]  Dinggang Shen, Guorong Wu, and Heung-Il Suk, “Deep Learning in Medical Image Analysis,” Annual Review of Biomedical Engineering, vol. 19, pp. 221-248, 2017.
[CrossRef] [Google Scholar] [Publisher Link]

[19] Jason Brownlee, “Machine Learning Mastery with Python,” Machine Learning Mastery, 2019.
[Google Scholar]

[20] Davide Chicco, and Giuseppe Jurman, “The Advantages of the Matthews Correlation Coefficient (MCC) over F1 Score and Accuracy in Binarys Classification Evaluation,” BMC Genomics, vol. 21, no. 1, pp. 1-13, 2020.
[CrossRef] [Google Scholar] [Publisher Link]

[21] Jesse Davis, and Mark Goadrich, “The Relationship Between Precision-Recall and ROC Curves,” Proceedings of the 23^rd International Conference on Machine Learning, Association for Computing Machinery, New York, NY, United States, pp. 233-240, 2006.
[CrossRef] [Google Scholar] [Publisher Link]

[22] Sebastian Raschka, “Model Evaluation, Model Selection, and Algorithm Selection in Machine Learning,” arXiv preprint, pp. 1-49, 2018.
[CrossRef] [Google Scholar] [Publisher Link]

[23] Pushpa Singh et al., Diagnosing of Disease using Machine Learning, Machine Learning and the Internet of Medical Things in Healthcare, Academic Press, pp. 89-111, 2021.
[CrossRef] [Google Scholar] [Publisher Link]

[24] Tae Kyun Kim, “T Test as a Parametric Statistic,” Korean Journal of Anesthesiology, vol. 68, no. 6, pp. 540-546, 2015.
         [CrossRef] [Google Scholar] [Publisher Link]