This question was raised by Marco in our discussion yesterday evening about the importance of speakers vs electronics. I didn't comment on it specifically at the time, but here's some thoughts about why we bother with hi-fi.
I think there's two sets of factors at work here; one sonic, the other not. The two are, IMO, inextricably linked.
The not-sonic factors first. We're almost exclusively blokes, so we like 'things'; some of like to take them apart and use a soldering iron on them, others just like reading spec sheets and watching the LEDs. This aspect of hifi is related to train-spotting, cameras, all sorts of 'collecting' hobbies, and DIY done from wish rather than poverty or necessity.
Now the sonic factors. As I said, I think all the talk of 'emotion' and 'musicality' is beside the point; both can be had from very basic musical reproduction devices, and many music lovers have very lo-fi systems.
The analogy I'd use is films. Many films 'work' on a small screen, and if a film is badly acted and/or poorly directed or scripted, it won't be any better on a big screen, though the visual and sonic impact might distract from the weaknesses.
In the same way, good music will shine through the medium; poor music won't be any better via hi-fi, but listening out for tiny details in the recording might hold the listener's attention for a while. However, just as some films come to life on a big screen, some music (eg opera, large-scale orchestral pieces) do really benefit from a more 'spacious' acoustic.