Agriculture productivity plays a very significant role in the Indian economy. It contributed more than 17-18% to our country’s GDP. Agriculture is the major occupation in India. The economy of our country highly depends on agriculture and its associated products. India stood first in the world with the highest net cropped area followed by US and China. Various pests and diseases affect the plant growth quantity and quality of the product. So it is very necessary to detect the disease at an early stage of the growth of the plant. Plant disease and pests identification are carried out by image processing. In recent years there is the use of machine learning, computer vision, and deep learning to detect and identify plant disease. These automated techniques are very beneficial to monitor large farms in less time period. In this research we worked with potato plant leaves to detect healthy leaf, early blight, and late blight diseases, and also calculated the infected area of the plant leaf. We used a customized convolutional neural network to classify the leaf disease and achieved 96.0% accuracy for the CNN model.