Answering my own question here... turns out there was a version mismatch. Oh dear 😞
To summarise...
Perform backups using pg_dumpall and hive them off the server somewhere. This gives you textual SQL files, if you prefer, pg_dump can do different things, but works on an individual database at a time.
I'd also recommend you setup your server using something like Ansible. It means that things like your properties file and other details will be the same next time you need to (re)install.
To restore:
Stop FusionAuth if it's running
Make certain FusionAuth is the same version as was used when the backup took place!
Go double-check those versions 😉
Restore by running sudo -u postgres psql < dump.sql
Restart FusionAuth (not in silent mode)
For More sensitive environments, you may want to consider database replication and making a warm-standby server.