The volume knob will stay in the same position (assuming amp sensitivity is the same, which is totally unrelated to power).
As you say, it's all in the reserves which are available for peaks.
I've had 300 - 500wpc amps - some of which I liked and some of which I didn't, just on sound quality grounds.
So if you love your 80wpc job, then it's a sound quality lottery as to whether you would prefer a 500wpc amp. It will sound different, but that will not be very strongly related to the power available.
Some speakers present difficult loads and would welcome the extra power (or, more likely, would welcome extra current delivery) and you would hear that as an improvement in sq, but by the sound of it you have a good amp/speaker match already.
As ever, it all depends ...