E leans a little toward B sharp, so to speak
I guess you mean E#/F not B#.

I wonder if want you are hearing is down to the spectral component in the output. The actual pitch that any sound from a DAC is locked to is (in most cases, not so much Async USB) the source clock and is not under the DAC's control.