The work in this paper designs an image-based rice disease detection framework that takes rice plant image as input and identifies the presence of BrownSpot disease in the image fed into the system. A CNN-based disease detection scheme performs the binary classification task on our custom dataset containing 2223 images of healthy and unhealthy classes under complex environments. Experimental results show that our system is able to achieve consistently satisfactory results in performing disease detection tasks. Furthermore, the CNN disease detection model compares with state-of-the-art works and procures an accuracy of 96.8%.