FusionAuth cluster setup
Using Clustered FusionAuth
FusionAuth is stateless and typically CPU bound. Clustering FusionAuth nodes improves performance and redundancy. You can run as many FusionAuth nodes in a cluster as you’d like.
FusionAuth stores almost all state in its database. For clustered environments, the following interactions occur between nodes:
When the system log is downloaded, all node’s logs are provided
FusionAuth can be run in multiple architectures; see the Server Layout documentation for more.
Before you cluster multiple servers or containers running FusionAuth, prepare your environment. In addition to FusionAuth, you’ll need the following components:
A database. This will be used by all nodes to maintain state. While you could have the database on a node running FusionAuth, you’ll see better performance on a dedicated database server.
A load balancer in front of the FusionAuth nodes. This will distribute traffic between them.
An Elasticsearch cluster, if using the Elasticsearch search engine.
This infrastructure must be created and managed when operating a FusionAuth cluster. However, this setup is beyond the scope of this document.
These instructions assume you have a load balancer, optional Elasticsearch server, and database server already configured.
When building a FusionAuth cluster, consider:
What load balancer will you use? Software or hardware load balancers both work. You can use a vendor managed balancer like an AWS application load balancer or an open source web server such as nginx.
By what algorithm will traffic be distributed? If all the nodes have equivalent capacity, a round robin algorithm is fine.
Where will you terminate SSL? Typically this is done at the load balancer, but can be done at the FusionAuth nodes.
What level of security and network isolation do you need? Build the architecture that suits your needs. You can run FusionAuth in a private network and have all HTTP connections proceed through the load balancer, with SSH connections happening through a jump box, for example.
What version of FusionAuth will you be running? All nodes must be on the same version for correct functionality.
How will you manage FusionAuth configuration? All nodes must have the same configuration or undetermined behavior will occur. Ensure that configuration changes are replicated to every node.
With a standalone database you must use the advanced database installation and run the database creation and migration scripts outside of FusionAuth. How will you manage this? Do you have in-house tools you can leverage to do so?
User the advanced database installation instructions to create and populate the FusionAuth database. Add a FusionAuth database user and password. Record the connection information; you’ll want a JDBC URL, the username and the password.
Install FusionAuth on each of the servers or containers which you plan to run. You can install the software via RPM, DEB, zip file or any of the installation methods.
Build your FusionAuth configuration. Double check the following settings:
fusionauth-app.urlshould typically be blank. You may need to manually specify this value if you have multiple FusionAuth nodes and the only way the nodes can communicate is on a public network. In that case, specify each node’s public address.
production. This setting ensures your users will never see maintenance mode. You will have to apply database upgrades manually. More here.
database.urlwith the full JDBC connection string URL recorded above.
database.usernameto the database user name recorded above.
database.passwordas the database password noted above.
Distribute your FusionAuth configuration to all nodes. You can do this by setting environment variables, Java system properties, or by pushing the
fusionauth.properties file to each server.
Restart the instances to ensure configuration changes are picked up.
Add the instance addresses to your load balancer. If you are terminating TLS at the load balancer, proxy the http port, otherwise communicate over the TLS port. Both of these are configurable, but they default to
Configure the load balancer to forward the following headers to FusionAuth:
X-Forwarded-Proto: typically this will be
https. This ensures any redirects are sent with the appropriate scheme.
X-Forwarded-Host: The original host requested by the client in the
HostHTTP request header.
X-Forwarded-For: The originating IP address of the client.
X-Forwarded-Server: The hostname of the proxy server.
You can see community submitted proxy configurations in the fusionauth-contrib repo.
If you have difficulty installing FusionAuth in a cluster, you can set up a cluster with one node. Set up your load balancer to point to only one server, and get this working before adding any other nodes. This will narrow down any issues you may encounter.
Verify that the installation is clustered by navigating to. You’ll see multiple nodes listed:
The node which served the request you made has a checkmark in the This node field.
Node 1 served the above request.
You may see incorrect IP addresses for each node if you are using a version of FusionAuth prior to 1.23. This bug doesn’t affect clustering functionality. All other information about the nodes is correct.
While ssh access to each node is helpful for initial installation and troubleshooting, you should not need it during normal cluster operation. Modify your firewall accordingly.
You may also lock down the FusionAuth nodes to only accept traffic from the load balancer, so that all HTTP traffic goes through it.
If your load balancer supports health checks, call the status API. A
GET request against the
/api/status endpoint will return a status code. It’ll either be
200 if the system is operating as expected or non
200 value if there are any issues with the node.
There’s an open issue to add a Prometheus endpoint but Prometheus is not currently supported.
Available since 1.16.0-RC1
Should you need to review system log files in the administrative user interface, you can see those by navigating to. Logs for all nodes are displayed there.
See the Troubleshooting documentation for more information about logs.
Adding and Removing Nodes
To add more nodes to the cluster, do the following:
Stand up new FusionAuth servers.
Provide the same FusionAuth configuration as the existing nodes. In particular, provide the same connection info for the database.
Update your load balancer to send traffic to the new node.
To remove nodes, simply:
Update your load balancer configuration; remove the node that you’ll be shutting down.
Stop FusionAuth on the node to be removed.
Verify that the node disappears from the node list displayed at.
Here’s a video covering how to add and remove nodes from a FusionAuth cluster:
How Many Instances Should I Run?
To determine the number of nodes to run, load test your cluster. Usage, installation and configuration differ across environments and load testing is the best method to determine the correct setup for your situation.
Any commercial or open source load testing tool will work. Alternatively, use the FusionAuth load testing scripts.
If you’d prefer detailed architecture or design guidance customized to your situation, please purchase a support contract.
To upgrade your cluster, first schedule a downtime window. During that window:
Take down all the nodes.
Upgrade the database schema by running the migration SQL scripts.
Upgrade FusionAuth on each node.
Start all the nodes.
The recommendation is to automate this process using whatever scripting or configuration management tool you like. This will minimize the downtime.
As a point of reference, for our hosted solutions FusionAuth uses a configuration management tool and typically sees downtime on the order of seconds.