In this paper, an effective human detection method in thermal imaging is proposed using background modeling and convolution neural network(CNN). For real-time implementation, the background modeling is done by modified running Gaussian average and the CNN-based human classification is performed for only detected foreground objects. To enhance human detection accuracy, morphological operators and ellipse testing are adopted to extract Region of Interest. Also, three CNN models with different input sizes and voting method are trained using our own dataset. For real-time system, the whole system is implemented in C++ and it process more than 30 fps with high accuracy.