The outputs are independently buffered so there’s no issue with having a different load connected to the XLR and RCA outputs. We see this setup often and in some cases it’s as you describe and in others both are used simultaneously (e.g. to drive the main system as well as a whole-home system).