Multimodal-based Pediatric Diseases Diagnosis

Abstract

In our project, we aimed to solve the pediatric diseases delayed or mistaken diagnosis cases. For this prototype, we concentrated on only five pediatric diseases classification as a proof of concept. The selected have confusing visual symptoms, they are Chickenpox, Kawasaki, Measles, Roseola and Scarlet fever. For the classification, we use both the images capturing the visual symptoms and textual symptoms representing the internal and external state of the patient. The image dataset was collected from public repositories, while the textual dataset was synthetically constructed. For images disease classification we used Residual Network (ResNet50) and for text disease classification we used Long Short-Term Memory (LSTM). The purpose behind using two models was to create one model by fusing them that can classify diseases based on two types of information. The results for the trained models ResNET50, LSTM and their fusion were good (over 90% of accuracy). Our final application is an android mobile application that uses the fused model to allow doctors to do prompt accurate diagnosis of pediatric diseases.