Machine tool vibrations cause uncomfortable noise, may damage the edges of cutting tools or certain parts of machine tools, but most importantly, they always have negative effect on the quality of the machined surface of workpieces. These vibrations are especially intricate in case of milling processes where complex tool geometries are used, like helical, serrated, non-uniform pitch angles, and so on. During the milling process, the arising vibrations include free, forced, selfexcited, and even parametrically forced vibrations together with their different combinations. Regarding surface quality, the most harmful is the self-excited one called chatter, which is related to the regenerative effect of the cutting process. Its relation to machined surface quality is demonstrated in an industrial case study. The modelling and the corresponding cutting stability are presented in case of a helical tool applied for milling with large axial immersions. The extremely rich spectrum of the measured vibration signals are analyzed by means of model-based predictions, and the results are compared with the spectral properties of the corresponding machined surfaces. The conclusions open the way for new kinds of chatter identification.