Road safety remains a global concern, with driver-related factors like distraction, drowsiness, and medical conditions such as stroke being leading causes of accidents. In this paper, we propose a real-time ensemble learning framework that leverages transfer learning for the detection of stroke, drowsiness, and distracted driving. Our model integrates multiple Convolutional Neural Networks (CNNs) fine-tuned for each specific task, and employs a stacking method to combine the predictions of these models using a meta-classifier. Notably, the model is optimized to enhance stroke detection, minimizing false negatives—an essential aspect for timely medical intervention. Experimental evaluations on diverse datasets demonstrate the efficacy of our approach, achieving an overall accuracy of 92.5%. The results emphasize the model’s potential for real-time driver monitoring, offering critical safety features that could reduce accidents and save lives.