My first question is a) when I setup the dashboard on server A. Can I assume that all the servers (behind a load balancer) will get the same configuration?
You need to distribute the configuration file or the environment variables yourself. You could scp the configuration file to each server, for example.
b) I want to make sure that FA is using the RDS database and not a local one in each docker container. How can I prove to myself that FA is using the RDS?
I'm not sure how you installed this, but you could remove postgresql from the docker-compose file. You can also shut down the RDS instance and see if FusionAuth fails.
You can also look for a line in the startup log file that looks like this:
fusionauth_1 | 2021-02-02 9:23:53.070 PM INFO com.inversoft.jdbc.hikari.DataSourceProvider - Connecting to PostgreSQL database at [jdbc:postgresql://db:5432/fusionauth]
This line is connecting to a local postgresql database, but you should see the configuration value pointing to the RDS hostname in yours.