Abstract: Oral cancer has emerged as a critical public health problem, with a rising incidence of cancer, thereby increasing mortality rates as a consequence. The diversity of cancer types, as well as the variability of different imaging modalities, has complicated the process of accurate diagnosis of cancer. This study aims at achieving this objective through the creation of a novel deep learning model. A hybrid model, named ResTransNet, was created by combining Residual Networks models with transformers, which are capable of accurately classifying images based on both local and global features of images. Bayesian optimization was used to optimize the hyperparameters of the model. The model was tested on two different datasets of histopathological images of diverse types of lesions occurring in the oral cavity, comparing the results with conventional Convolutional Neural Networks models as well as Vision Transformers. The ResTransNet model was able to achieve a classification accuracy of 96.21% on the first dataset, as well as 92.11% on the second dataset, thereby proving its efficacy over conventional models. The ResTransNet model has the potential to become a strong diagnostic tool in the accurate diagnosis of cancer, owing to its hybrid ability to improve the process of image classification.

Key words: Oral cancer classification')">Oral cancer classification, Histopathological image analysis, Residual neural networks, Vision transformers, Hyperparameter optimization