Rice, being a widely grown food crop, often suffers from nutrient imbalances, hindering its production, which suffers from a lack of certain nutrients like potassium, nitrogen, and phosphorus. Identifying the potential nutrient deficiency in the rice crop can pose a challenging task because the leaves of rice plants affected by these nutritional shortages may have disorders such as colour and shape changes. The classification of nutrient deficits can be done visually by examining the colour and shape of the leaves. However, it is expensive, time-consuming, and requires greater expertise. To tackle this issue, this work proposes a computer vision-based deep learning approach called CAR-Capsnet, an enhanced version of the capsule network (Capsnet) that utilises contextual attention routing (CAR) for classifying nutrient deficiencies in rice crops. The dataset required for training and evaluation is obtained from the freely available Kaggle data repository. Pre-processing techniques, such as the Wiener filter and adaptive Otsu segmentation, are applied to the dataset. Unlike traditional routing approaches, CAR-Capsnet utilises contextual attention routing to improve the model’s capability to handle complex visual features and patterns associated with nutrient deficiencies in rice crops. In the evaluation, the proposed model’s performance is compared with two other existing models, CNN and the original Capsnet. Experimental results demonstrate that CAR-Capsnet outperforms both models, achieving a testing accuracy of 97.1% and exhibiting superior performance compared to the baseline models. Additionally, comparing the classifier’s results with previous research findings confirms that the proposed CAR-Capsnet model is superior to alternative approaches in classifying nutrient deficiencies in rice crops.