The real problem is that if your outside internet goes down, you can’t log into your own server.

That’s not true. You can specify IP addresses that are allowed to connect without authentication, eg your local network. So in the event of an internet outage you can still watch tv.