This study investigates the latent factors underlying reading anxiety in English language classes and also measures the factors’ relationships to both language proficiency and reading competence, focusing on Korean college students. A total of 313 learners from a variety of majors were recruited for this study. Two main instruments were used: a background questionnaire and the English Language Reading Anxiety Scale (ELRAS). Exploratory factor analysis and the Pearson product-moment correlation were used to identify possible causes of reading anxiety and their correlations to language proficiency. The findings confirm a strong validity for the ELRAS instrument, and from the data extracted, six factors are shown to be anxiety-inducing: anxiety when reading English, difficulty using cognitive reading strategies, lack of interest towards English-speaking countries, English proficiency and reading confidence, English reading environments, and teaching methods used during reading lessons. This study also reveals significantly negative relationships between reading anxiety and language abilities, and reading competence. Based on the findings, practical teaching methodologies and recommendations for language classrooms are offered to help teachers better nurture their students’ reading development.