Found the answer in the manual:
So YES, we can safely link pins 1 & 3 to be the ground for the RCA output.