In this paper, we first explore a modified adaptive Gaussian mixture model (MAGMM) by investigating the confidence value of observation vectors to deal with noise conditions problem. The observation confidence values are estimated by using the frame SNR values calculated between the input noisy speech and the enhanced speech, and the sigmoid function. We compare three speech enhancement techniques, minimum mean square error logarithm short- time spectral amplitude (MMSE log-STSA), low-rank matrix recovery (LRR) and multiple low-rank representation (MLRR), for observation confidence computation. Furthermore, we also consider the effect of the use of observation confidence value in the GMM-supervector (GSV) and i-vector approaches which are considers as input feature vectors for the Support vector machine (SVM). To verify the accuracy of the speaker system, we use utterances from a Korean drama “You came from the stars.” The experimental results show that our proposed approaches achieve better performance than the baseline systems under noisy environments.