Visual Question Answering Support to Arabic Pedagogical Tool

Abstract

The Arabic language serves as a highly valuable strategic global language due to its widespread use. However, learning Arabic as a second language is quite a challenging task due to the specificity of Arabic, which encompasses many major challenges: Semitic language with complex grammar, high dialectal variation, rich phonetic specter within a vast vocabulary that includes many words with similar meanings. Faced to the high scarcity of dedicated pedagogical tools, we address this problem by designing an end-toend web-based pedagogical tool leveraged by Deep Learning. In fact, our convivial and ergonomic web-based tool, is based on constructivism learning model where the learning process is made through active learning. In our tool, we offer a high quality pedagogical activities for Arabic Learners such as real-life vision quizzes, interactive image-based questions, and language learning activities. These activities are designed to enhance the learning experience by integrating visual and language processing tasks, providing a comprehensive approach to language education. Within the tool, we deployed and harnessed at least two recent AI deep learning based models: Text-To-Text Transfer Transformer which is a large language model. This later is the basis for translation and Vision Question Generation Vision Language Pre-training which is a large-scale generalized model that has been trained on multiple tasks using a large volume of data and then fine-tuned on tasks like Vision question answering. This Model is deployed for Visual Question Answering. In addition, we have designed and implemented the tool front-end while relaying on SOTA technologies React.js, React Query, Tailwind CSS, In order to measure the performance of our AI-based pedagogical tool, we conducted a human evaluation using 150 vision quizzes. The results show that our tool is very suitable for Arabic Learning with an accuracy of 80%.