Abstract: Fine-grained image classification has high research value and application prospects in practical applications. At present, the traditional lithology fine-grained classification method is highly subjective and time-sensitive, depending on the experience of researcher and the quality of experimental equipment. Therefore, in this paper, the technology of fine-grained image classification is introduced into the field of granite lithology identification. The RGB image datasets of four types of lithology, namely flesh-red, grayish-white, iron-manganese, and amazonite-bearing alkali feldspar granites, are systematically constructed. Comparative experiments were carried out using typical deep learning models such as AlexNet, VGG16, ResNet50 and Vision Transformer. The results show that the classification accuracy of all models exceeds 82 %, and the VGG16 model is the best, which is 88.57 %, an increase of 5.71 % over the AlexNet model; the amazonite-bearing alkali feldspar granite has a recognition accuracy of 100 % due to its significant characteristic minerals, while the grayish-white alkali-feldspar granite has the worst recognition effect; the model accuracy is positively correlated with the amount of training samples, and the model performance is optimal when the training set is complete. In the future, the accuracy of fine-grained classification of rare metal granite lithology can be further improved by improving the quantity and quality of rock samples and optimizing the model algorithm.