A phenomenon called rotating instability was rarely reported in a centrifugal compressor, although some associated discussions can be found in axial compressors. This paper presents a numerical investigation on the unsteadiness of the tip leakage flow and rotating instability in an isolated centrifugal compressor impeller. A three-dimensional, unsteady full annuls simulation is performed under three different operating conditions. The tip leakage flow unsteadiness at the impeller inlet and inside the impeller passage is detected by the pressure fluctuation. During the throttling process, the temporal and spatial propagation characteristic of the tip leakage flow unsteadiness under the near stall operating condition is revealed based on the wavelet analysis, Fourier transform, and dynamic mode decomposition. Furthermore, the relationship between the tip leakage flow unsteadiness and the rotating instability is also discussed. The results show that the pressure fluctuation mainly concentrates in the tip region at the near stall operating condition, which indicates the occurrence and circumferential propagation of tip leakage flow unsteadiness. The circumferential propagation of the tip leakage flow unsteadiness induces a significant circumferential rotating pressure wave with a wave number of 9 and approximately 48.2% of the rotational speed of the impeller. The characteristic frequency of the tip leakage flow unsteadiness agrees well with the dominant mode frequency of rotating instability. Therefore, the tip leakage flow unsteadiness is assumed to be responsible for the rotating instability. During the throttling process, the formation and development of the rotating instability at near stall operating condition can be divided into three stages based on signal analysis: prior to rotating instability, generation and development, and rotating instability.