The tape output of your pre-amp may put out a lower signal level than what comes in. At least, that wouldn't be impossible.
If the passive replaced your active pre-amp and it simply routed the signal from the DAC out to the headphone amp, no it wouldn't be a solution. A pair of attenuators world be a solution.
BTW, I notice the Asgard 2 has a high/low gain switch on the back. Does yours? If so, have you got it set to low gain?