The purpose of this research is to manage the physical sectors systematically, needed for the selection and direction of declining areas, such as urban regeneration and revitalization projects, by grading the density of decline indicators of buildings in small and medium cities. In five small and medium-sized cities in Jeollanam-do, the data were divided into three stages according to the density, according to the use, structure, and age of the building. Rating by element of Census Output Area, detailed designation, and management of declining areas in small and medium-sized cities are required, rather than setting the scope of declining areas based on administrative districts. It is also necessary to manage cities through complex and intra-element phasing rather than single elements in defining physical decline. The analysis based on the complex factors and boundary of the Census Output Area is meaningful in order to maintain and manage such cities in a more detailed and precise manner according to such characteristics of each region.