Multimodal post-operative complication prediction for elderly patients with hip fractures

Hip fractures are common among the elderly and they have become a major health care problem for society. Standard procedure is to have surgery, often causing complications that can lead to short-term mortality. With the help of an early warning system, we could take precautions to mitigate the consequences of these complications. Machine learning can be used to develop such a system. In this paper, we develop a multimodal deep learning model for post-operative complication prediction using both pre-operative and per-operative data from elderly hip fracture patients. We use ResNet50 to extract features from image modalities and employ LSTM units to extract features from per-operative vital signs. Features from different modalities are combined through early fusion of the features forming a single multimodal prediction model. Further, we also investigate the effect of each modality on the prediction task using SHAP. We evaluate our approach on an in-house data set with 1669 patients. We find that i) our model can predict short-term mortality and heart failure reasonably well and ii) the inclusion of per-operative features does not improve performance of the multimodal model. We use Shapley values to provide local and global explanations for our prediction task.