Is the problem the same if you connect the TOSLink directly to the Vivaldi DAC and configure Sync mode on the DAC to “Audio”?
If the problem is the same in both cases - connecting to the DAC directly vs. connecting to Upsampler+DAC (with proper clock configuration in both cases) - then the issue is likely to be with the source. 
Since the Nintendo Switch/Dock does not have a native TOSLink interface, where is that TOSLink coming from? What Adapter (brand/model) are you using? You might need to get a better one that properly conforms to the S/PDIF, TOSLink specification (IEC 60958)