Dynamic modes of the non-classical damping or very closely distributed modal system are nearly decomposed by using the conventional modal matrix calculated from mass, stiffness matrix because that the damping matrix is unable to be diagonalized with the modal matrix. In this paper, a new mode decomposition technique for a non-classical damping structure in frequency domain is proposed that is using the differential state variables estimated from measured acceleration responses and Kalman filtering to calculate the linear transformation matrix of governing modes. The linear transformation matrix is updated in the optimization process to maximize the performance index cooperated with a power spectral density of a target mode and an averaged power spectrum. For the verification of the proposed method, a numerical simulation for the representative non-classical damping and closely distributed modal system : primary mass-TMD coupled 2DOF system. The results from the simulations show that the proposed technique can estimate the modal response more precisely than a conventional mode decomposition method such as ICA, POD.