Did you ever consider to use an Audio Switch, such as this one?
They offer XLR and RCA versions. If you use single-ended for your main system, and also have a single-ended headamp, it looks like the way to go.