@dan unfortunately not yet in production due to issues not depending to me... but the cluster seems working... I also succeed to perform a software upgrade to the latest fusionauth version.

@dan unfortunately not yet in production due to issues not depending to me... but the cluster seems working... I also succeed to perform a software upgrade to the latest fusionauth version.

@dan thank you!
I actually tried starting the installing procedure in production mode but i had some issues...

Anyway i now have a working 2-node cluster in devel mode (on your input, i just checked /usr/local/fusionauth/config/fusionauth.properties files on both nodes, they are md5sum-equal so far).
For "working cluster" i mean that i see "2 nodes" in the system>about page

Is there any way to perform a more comprehensive test to check that everything is configured correctly...?

I will switch asap in production mode (that's my target anyway).

Btw, I found a lot simpler to perform initial configuration with only one node and then add the other one later with your Youtube guide.
Referring specifically to my cluster architecture i also had to:

• get rid of PgBouncer, it messes up with hikari and also with pgsql prepared statements (switching to session pool_mode in pgbouncer did not help)

• tweak haproxy a bit, especially to make it work with hikari... i am still not yet satisfied with the results and i will probably have to tweak also fusionauth/hikari db connections management

Coming to the load balancer configuration, the guide says to setup these header rules:
X-Forwarded-Proto: typically this will be https. This ensures any redirects are sent with the appropriate scheme.
X-Forwarded-Host: The original host requested by the client in the Host HTTP request header.
X-Forwarded-For: The originating IP address of the client.
X-Forwarded-Server: The hostname of the proxy server.

I more than able to setup a nginx ha setup to do that.

However, my cloud environnment provides out-of-the-box ha load balancers that work with these rules (not modifiable):
X-Forwarded-For - contains the IP address of the original client making the request
X-Forwarded-Port - contains the listen port of the Load Balancer service. So if your Load Balancer listens on port 80, it will contain 80.
X-Forwarded-Proto - contains the protocol the client used to contact the Load Balancer. Can be http or https.
I also checked that incoming Host header contains the load balancer's dns name.

Things seem running smoothly also with this last setup (please note that so far i just did configuration tasks and i did not use any apis or production features...)... can i proceed with this load balancer setup or am i going to face problems in production?

Thanks!

Hello.
I am trying to install fusionauth in HA deployment.

I'll give you the gist of my scenario (the 2nd one here: https://fusionauth.io/docs/v1/tech/installation-guide/server-layout/

• a virtualization environnment with a backend network (172.16.50.0/24) and some public IPs
• a three-nodes PGSQL Repmgr cluster (primary/standby/witness) only accessibile from the backend network
• a pgbouncer->haproxy->pgbouncer stack ( pgbouncer->haproxy on the client vms, the last pgbouncer on the pgsql vm)
• 2 client Debian11 VMs multihomed (public ip and backend access, but only the latter is used) with fusionpbx-search and fusionpbx-app 1.32.1 installed with debs.
• a nginx loadbalancer (with public/backend ip) with configuration partially taken from here: https://github.com/FusionAuth/fusionauth-contrib/blob/master/Reverse Proxy Configurations/nginx/nginx.cluster.conf

Both Pgbouncers are configured with pool_mode=session and ignore_startup_parameters = extra_float_digits

The installation goes quite smooth till i get to start both fusionapp on both nodes.... it seems the first one it's the one that registers itself on the db (in the nodes table), but the second one does not get there...

Log from the "first starter":

2021-12-26 5:58:30.843 PM INFO  io.fusionauth.api.configuration.DefaultFusionAuthConfiguration - Loading FusionAuth configuration file [/usr/local/fusionauth/config/fusionauth.properties]
2021-12-26 5:58:30.867 PM INFO  io.fusionauth.api.configuration.DefaultFusionAuthConfiguration - Dynamically set property [fusionauth-app.url] set to [http://172.16.50.19:9011]
2021-12-26 5:58:30.870 PM INFO  com.inversoft.configuration.BasePropertiesFileInversoftConfiguration -
  - Overriding default value of property [fusionauth-app.runtime-mode] with value [development]
  - Overriding default value of property [search.type] with value [elasticsearch]

2021-12-26 5:58:30.969 PM INFO  io.fusionauth.api.configuration.DefaultFusionAuthConfiguration - Loading FusionAuth configuration file [/usr/local/fusionauth/config/fusionauth.properties]
2021-12-26 5:58:30.969 PM INFO  io.fusionauth.api.configuration.DefaultFusionAuthConfiguration - Dynamically set property [fusionauth-app.url] set to [http://172.16.50.19:9011]
2021-12-26 5:58:30.971 PM INFO  com.inversoft.configuration.BasePropertiesFileInversoftConfiguration -
  - Overriding default value of property [fusionauth-app.runtime-mode] with value [development]
  - Overriding default value of property [search.type] with value [elasticsearch]

2021-12-26 5:58:31.709 PM INFO  io.fusionauth.api.configuration.DefaultFusionAuthConfiguration - Loading FusionAuth configuration file [/usr/local/fusionauth/config/fusionauth.properties]
2021-12-26 5:58:31.709 PM INFO  io.fusionauth.api.configuration.DefaultFusionAuthConfiguration - Dynamically set property [fusionauth-app.url] set to [http://172.16.50.19:9011]
2021-12-26 5:58:31.710 PM INFO  com.inversoft.configuration.BasePropertiesFileInversoftConfiguration -
  - Overriding default value of property [fusionauth-app.runtime-mode] with value [development]
  - Overriding default value of property [search.type] with value [elasticsearch]

2021-12-26 5:58:31.711 PM INFO  com.inversoft.maintenance.MaintenanceModePoller - Poller started to Wait for configuration to be completed.
2021-12-26 5:58:31.848 PM INFO  com.inversoft.maintenance.db.JDBCMaintenanceModeDatabaseService - [/aBhYQ] Database Version [1.32.1]
2021-12-26 5:58:31.848 PM INFO  com.inversoft.maintenance.db.JDBCMaintenanceModeDatabaseService - [/aBhYQ] Latest Migration Version [1.32.0]
2021-12-26 5:58:31.854 PM INFO  com.inversoft.search.ElasticRestClientHelper - Connecting to Elasticsearch at [http://localhost:9021]
2021-12-26 5:58:32.201 PM INFO  com.inversoft.maintenance.search.ElasticsearchMaintenanceModeSearchService - Missing search index [fusionauth_user]
2021-12-26 5:58:32.205 PM INFO  com.inversoft.maintenance.search.ElasticsearchMaintenanceModeSearchService - Missing search index [fusionauth_entity]
2021-12-26 5:58:32.208 PM INFO  org.primeframework.mvc.servlet.PrimeServletContextListener - Initializing Prime
2021-12-26 5:58:32.226 PM INFO  i.f.app.maintenance.guice.FusionAuthInteractiveMaintenanceModeModule -

---------------------------------------------------------------------------------------------------------
--------------------------------------- Entering Maintenance Mode ---------------------------------------
---------------------------------------------------------------------------------------------------------

2021-12-26 5:58:32.354 PM INFO  io.fusionauth.api.configuration.DefaultFusionAuthConfiguration - Loading FusionAuth configuration file [/usr/local/fusionauth/config/fusionauth.properties]
2021-12-26 5:58:32.355 PM INFO  io.fusionauth.api.configuration.DefaultFusionAuthConfiguration - Dynamically set property [fusionauth-app.url] set to [http://172.16.50.19:9011]
2021-12-26 5:58:32.355 PM INFO  com.inversoft.configuration.BasePropertiesFileInversoftConfiguration -
  - Overriding default value of property [fusionauth-app.runtime-mode] with value [development]
  - Overriding default value of property [search.type] with value [elasticsearch]

2021-12-26 5:58:32.355 PM INFO  com.inversoft.maintenance.MaintenanceModePoller - Poller started to Wait for configuration to be completed.
2021-12-26 6:02:55.759 PM INFO  com.inversoft.search.ElasticRestClientHelper - Connecting to Elasticsearch at [http://localhost:9021]
2021-12-26 6:02:57.042 PM INFO  com.inversoft.search.ElasticSearchClient - Created Elasticsearch index [fusionauth_user] successfully
2021-12-26 6:02:57.371 PM INFO  com.inversoft.search.ElasticSearchClient - Created Elasticsearch index [fusionauth_entity] successfully
:arrow_forward: 
2021-12-26 6:02:57.435 PM INFO  io.fusionauth.app.guice.FusionAuthModule -

---------------------------------------------------------------------------------------------------------
--------------------------------- Starting FusionAuth version [1.32.1] ----------------------------------
---------------------------------------------------------------------------------------------------------

2021-12-26 6:02:57.617 PM INFO  io.fusionauth.api.plugin.guice.PluginModule - No plugins found
2021-12-26 6:02:57.752 PM INFO  io.fusionauth.api.service.system.NodeService - Node [c47264a3-a560-456a-af8c-908767007f88] started.
2021-12-26 6:02:58.367 PM INFO  io.fusionauth.api.configuration.DefaultFusionAuthConfiguration - Loading FusionAuth configuration file [/usr/local/fusionauth/config/fusionauth.properties]
2021-12-26 6:02:58.368 PM INFO  io.fusionauth.api.configuration.DefaultFusionAuthConfiguration - Dynamically set property [fusionauth-app.url] set to [http://172.16.50.19:9011]
2021-12-26 6:02:58.368 PM INFO  com.inversoft.configuration.BasePropertiesFileInversoftConfiguration -
  - Overriding default value of property [fusionauth-app.runtime-mode] with value [development]
  - Overriding default value of property [search.type] with value [elasticsearch]

2021-12-26 6:02:58.370 PM INFO  com.inversoft.jdbc.hikari.DataSourceProvider - Connecting to PostgreSQL database at [jdbc:postgresql://localhost:6432/fusionauth]
2021-12-26 6:02:58.372 PM WARN  com.zaxxer.hikari.HikariConfig - HikariPool-1 - idleTimeout has been set but has no effect because the pool is operating as a fixed size pool.
2021-12-26 6:02:58.377 PM INFO  com.zaxxer.hikari.HikariDataSource - HikariPool-1 - Starting...
2021-12-26 6:02:58.422 PM INFO  com.zaxxer.hikari.HikariDataSource - HikariPool-1 - Start completed.
2021-12-26 6:02:59.502 PM INFO  com.inversoft.scheduler.DefaultScheduler - Starting up scheduler
2021-12-26 6:02:59.503 PM INFO  com.inversoft.scheduler.DefaultScheduler - Scheduler is running
2021-12-26 6:02:59.655 PM INFO  io.fusionauth.api.service.system.NodeService - Node [c47264a3-a560-456a-af8c-908767007f88] added with address [http://172.16.50.19:9011]
2021-12-26 6:02:59.714 PM INFO  com.inversoft.search.ElasticRestClientHelper - Connecting to Elasticsearch at [http://localhost:9021]
2021-12-26 6:05:53.339 PM INFO  io.fusionauth.api.service.system.NodeService - Node [c47264a3-a560-456a-af8c-908767007f88] promoted to master at [2021-12-26T17:05:53.338930019Z]

The log on the "second starter" has similar log but it stops at

I can access via web to the cluster, and i get to create elasticsearch indexes.. but after that it starts to loop like this (nginx access log):

87.7.xxx.127 - - [26/Dec/2021:18:03:43 +0100] "GET /maintenance-mode-workflow HTTP/2.0" 302 0 "https://auth.xxx.cloud/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/96.0.4664.110 Safari/537.36" "-"
87.7.xxx.127 - - [26/Dec/2021:18:03:43 +0100] "GET / HTTP/2.0" 302 0 "https://auth.xxx.cloud/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/96.0.4664.110 Safari/537.36" "-"
87.7.xxx.127 - - [26/Dec/2021:18:03:43 +0100] "GET /maintenance-mode-workflow HTTP/2.0" 302 0 "https://auth.xxx.cloud/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/96.0.4664.110 Safari/537.36" "-"
87.7.xxx.127 - - [26/Dec/2021:18:03:43 +0100] "GET / HTTP/2.0" 302 0 "https://auth.xxx.cloud/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/96.0.4664.110 Safari/537.36" "-"

After a few minutes i get a lot of:

PSQLException: ERROR: prepared statement "S_*" already exists

on my fusionauth app (on the first node):

Could be an issue of my db setup?
Can anyone give me an hint?

Thanks