Among indoor and outdoor environments, ambient lighting can greatly affect a face image. Due to the face texture, a direct lighting can cast strong or light shadows that diminish or accentuate some facial features in some applications such as face recognition. For that reason, we propose an approach to detect shadows and compensate their effects in these applications. While the shadow can be detected based on within-class variance, the compensation is to correct the luminance and chrominance of each pixel using the adjacent pixels and their own information. We construct two compensation functions. One is for luminance compensation; the other is for chrominance compensation. The experimental result shows that the compensated face image has reduced the shadow effects without generating many visual artifacts. The applied algorithm is effective and robust under varying illumination.