Nick, more often than not the problem has to do with the iPad/iOS disconnecting from your WiFi network when it locks from inactivity; iOS is kind of all over the place across versions on how WiFi works exactly.
Assuming your home WiFi network isn’t the issue, one recommendation is to turn OFF “Auto-Lock” (Settings → Battery → Auto-Lock → Never), and just explicitly turn off the iPad only when you’re no longer listening. That worked for me in the past when I had similar issues.
There are other settings that seem to be required in some instances, for example, Privacy → Location Services → System Services → Networking & Wireless to OFF, but I haven’t personally had to do any of that with iOS 13.5 (or prior).
Good luck 
and let us know how it goes.