I think it largely depends on what level of sound you are aiming for. There are many good value for money components that if chosen wisely would make a decent sound together for not a lot of money. However, if you are aiming rather higher you need to think a little differently. For me, as a vinyl lover, the turntable has to be first, making sure it has a good motor to turn at the right speed, & a good bearing for consistency. This then has to be partnered with a good arm, and if these are well chosen then even a budget cartridge will sound good, and can be improved upon later is money is tight. I have always thought that the phono stage is very important, in fact I spent more money on my PS than I did my amp, and went through about 8 of them in just over a year before I settled on what I have now. It is extremely flexible as well which means in my case i don't need a separate preamp, and has variable gain so I can match it with pretty much any cartridge I want.
I have found, from personal experience, and especially if you buy used, the amp to matter not so much. I have tried many in my system recently and while none have bettered my own, to my ears at least, the sound from all of them was easily good enough to live with if I had to. Lastly the speakers, but certainly not last in importance, and again I have tried many in a short space of time, they not only have to sound good in their own right, and with your gear, but I would say interaction with the room is the biggest thing to get right.