Natural lighting condition has been the limitation of automatic lip detection in the lip reading. To cope with the problem, we present a robust lip detection algorithm applicable to lip reading in the smartphone environments. Our proposed lip detection approach utilizes the partial color based clustering, clustering with geometric feature and fuzzy confidence based verification. The partial clustering into multiple clusters based on color information can model the skin, lips and other regions sufficiently and enables to avoid the effect of illumination on the lip images. The geometry based clustering is used to efficiently separate lips from other regions and combine the discrete components of one facial feature. The lip confidence using fuzzy rules is introduced to select the best segmented lip candidate. To validate the proposed algorithm, we constructed database in standard, indoor, outdoor environment. The experimental results show that in each environment the lip detection performance of our proposed method enhanced at abut 7%, 13%, 26% than conventional approaches. Through the experiments, we have shown the robustness against illumination and lightweight performance of our proposed algorithm compared to the conventional approaches.