There’s not really an easy or simple answer to this question…
The first point to make is that balanced connections are all about reducing the amount of noise picked up between the two units. In that sense, whether it’s balanced or not is immaterial to the sound quality where the cables lengths are modest, the physical environment isn’t harsh and the amount of noise picked up should be low.
Given that, one could surmise that a balanced connection is going to be detrimental to the sound quality as the balancing/unbalancing might be done by a transformer, which is definitely not an audio neutral device, or else if it’s an electronically balanced circuit, there are twice the number of amplifiers in the output stage and they have to be carefully matched. There may also be some cancellation of even harmonics generated in the output stages (which could be deemed as being positive or negative) if the balanced audio output isn’t given the attention it deserves - but we feel that we do a very good balanced output which should be up there with the very best.
In our implementations, the balanced output impedance is much lower than that of the unbalanced. In THEORY this means that cable (etc.) capacitance will tend to roll off the frequency response sooner with the unbalanced than the balanced output stage however in practice even an implausibly capacitive cable would only start to roll off things above 1MHz, so it’s hard to imagine that having much effect in reality given the general response of amplifiers, speakers and ears in the MHz+ range.
The final factor could be the input stage of the destination equipment which is something we have no control over and which can be expected to vary a lot from one piece to the next. Implementing an unbalancing stage with valves is quite involved. Sometimes this is skipped and the XLR “hot” connection is simply connected to an unbalanced input (grounding or leaving open the unused leg). Some may use a transformer and transformers are typically not “audio neutral” components (as above). They may well add distortion at low frequencies and roll off high frequencies, the former of these can definitely have euphonic properties so it could well be possible that some listeners who express a preference the “sound” of the balanced output simply prefer the audio signature of the balanced input of whatever they’re driving.
So, to sum up there’s no easy answer and lots of variables, many of which are beyond our control but there is no intrinsic reason that one connection will “sound” better than the other…
Myself, I use the RCA outs on my dCS kit here at home into the unbalanced (but XLR) inputs on one of my NAP300’s and I’m really happy with the sound apart from I wish it went louder, I do have a NAP500 but I prefer the sound of my NAP300.
(Yes, Naim use XLR’s on the NAP250, NAP300 and NAP500 but for UNBALANCED audio, not balanced audio.)
I hope this helps…