Even though numerous target tracking algorithms have been developed and proven successful, performance is still negatively impacted by occlusion and lighting changes. In this research, a successful tracking technique for non-rigid objects in video situations is described. The use of computer vision to address real-time violent activity is offered as an intelligent system algorithm. Different acts of violence occasionally occur in our daily lives when we are absent. Real-time violent activity detection is essential to a smart surveillance system. Since a video consists of several frames of pixels, classifying and analysing the video is a difficult investigation problem in the field of artificial intelligence. Convolutional Long Short Term Memory is utilized in this research to take into account every scenario that could arise in order to recognise real-life violence more correctly. Datasets gathered from diverse sources are compared to get a result with a sufficient level of accuracy. With multiple experiments employing various deep video analysis algorithms, the research effort came to a conclusion. The several deep learning models are compared, and the top candidate with a 96% accuracy rate is chosen. Real-time video has now been set up to categorise anomalous activity using the proposed model.