No problem. It's not a disaster if that's where you end up using the dials, but the Goldpoint's been designed to sound best, used with them turned down more (to achieve the same loudness level), as at that point there are less shunt resistors in the signal path.
You have both shunt and series resistors in the Goldpoint. The Z-Foils I mentioned earlier are the series resistors, and there are only two (one for each channel, left and right).
Essentially, the more you turn up the volume (increase each 'step' on the dual-mono attenuators), the more shunt resistors are 'in circuit', so the potential for sonic degradation increases with every 'click'. It's just the way it's been designed.
You can see all the (surface mounted) shunt resistors employed, with each attenuator 'step' (the little square numbered things), on this link:
http://www.goldpt.com/big.html
So you want to minimise the amount of steps you need to use, in order to achieve the desired volume.
The only way you'll be able to do that yourself, is to alter the current 'gain structure' in your system, either by using a source component with a higher output voltage, or a more sensitive power amp (or a combo of both), and then you won't need to turn the Goldpoint up as much.
You could also alter the value of the series resistors, to achieve a similar effect, but that's a more technical thing and involves some soldering. Hope that all makes sense!
This is also a useful link, explaining the differences (pros and cons) between series, ladder and shunt-type stepped attenuators, used in passive preamps:
http://www.goldpt.com/attenuator_types.html The Goldpoint you're using is a series-type.
Marco.