Exactly!!!!!!!!!!!!

The phase noise of the reference will be reduced by 20*log N, where N is the integer division rate. If the division ratio is 1000 (servo running at 400 Hz), you have picked up 80 dB of noise reduction. You are now into the noise floor.

5 PPM of short-term drift might be 20 Hz, at 4 MHz. But it is now 0.002 Hz, at 400 Hz. Do you think you will hear that much speed variation?