Once you’ve installed FusionAuth and integrated it with your applications, you have to run it. For that you need insight into how the system is performing. There are a number of ways to gain this insight.
Monitoring Summary Table
Here’s an overview table of monitoring options.
|Name||Contains||Ingestion Options||More Information|
|System Logs||Exceptions, stack traces, database connection issues, Elasticsearch connection issues. Minimal tracing capability unless debug logging is enabled.||Can be exported as a zip file via API.||More info|
|Audit Logs||Created by the admin user interface actions. This is just an API, so a customer can also call this API. This is a free form API, and the audit log contents are whatever you put in.||This can be consumed via web hook or API.||More info|
|Event Logs||Debug information for external integrations with IdPs, SMTP etc. In general, runtime errors that are typically not caused by FusionAuth and cannot be communicated well at runtime. Examples: a template rendering error in a custom theme, an exception connecting to SMTP due to bad credentials, a failure in a SAML exchange, a connection to a webhook is failing.||This can be consumed via web hook or API.||More info|
|Login records||Each successful login is recorded here. Includes IP information, application, user, and timestamp.||Can be consumed via API. This record itself is not sent through a webhook, but a login success or login failure can be consumed via a web hook.||More info|
|Webhooks||Triggered by events as documented, sending is configurable. Contains IP and location information when available.||Can be sent to an HTTP endpoint or a Kafka topic.||More info|
|Metrics||System metrics, such as JVM buffers, memory and threads.||Can be consumed via API and available in Prometheus compatible format.||More info|
The system log files will be placed in the
logs directory under the FusionAuth installation unless you are running FusionAuth in a container. In the latter case, the log output will be sent to STDOUT by default.
You may also set up a Docker logging driver to direct log files elsewhere.
System logs running in non-containerized instances can also be exported via the Export System Logs API.
Learn more about FusionAuth log files.
There are a few different APIs exposing FusionAuth application specific information you may want to ingest into your monitoring system:
- FusionAuth administrative user interface audit logs
- Logs and errors from asynchronous code execution
- Login records
In general these are APIs you will have to poll to ingest.
Log File Formats
Log File Information
|Log Type||Export Format||Timezone||Date Format||API Docs|
|Audit Logs||Zipped CSV||Specified by the ||Specified by the ||API|
|Login Records||Zipped CSV||Specified by the ||Specified by the ||API|
|System Logs||Zipped directory of files. Log entries separated by newlines, but may be unstructured (stack traces, etc).||For log entries, the timezone of the server. The ||API|
There are a number of events, such as a failed authentication or an account deletion, that you may want to ingest into your monitoring system. These are available as webhooks.
Here’s the list of all available events.
You can pull system metrics from the System API. The format of these metrics are evolving and thus are not documented.
You can also enable JMX metrics as outlined in the Troubleshooting documentation.
The default system metrics are also available in a Prometheus compatible form. This tutorial explains how to set up Prometheus to scrape the FusionAuth metrics.
FusionAuth is a Java application. You can use the OpenTelemetry project to instrument it. Doing so can assist troubleshooting performance issues and make it easier to run.
FusionAuth has no custom tracing spans embedded, but it can be set up as an OpenTelemetry exporter. Doing so provides useful data about types of requests, database queries, and more.
Set Up the Exporter
Download the Java OpenTelemetry agent. Make it available to the FusionAuth instance. If you are using a zip/deb/rpm, put it in a directory. If using docker, mount the host directory to the container.
Set an environment variable
OT_PATH to point to the directory.
Next, you need to configure FusionAuth to use the agent by passing the correct Java command line switches. There are a couple of ways to do this.
You can use Java system properties, environment variables, or edit the
fusionauth.properties config file. See the configuration reference for more info. For this example, environment variables will be used.
Set this environment variable and make sure it is available when you start FusionAuth (in a startup script,
Enabling OpenTelemetry export
Next, configure OpenTelemetry to point to your collector. Here’s an example of the environment variables to set to use HoneyComb as your collector. The specific variables needed will vary based on your chosen solution; please consult the collector’s documentation.
Configuring OpenTelemetry collector
# from https://docs.honeycomb.io/getting-data-in/opentelemetry/java-distro/#using-opentelemetry-without-the-honeycomb-distribution export OTEL_EXPORTER_OTLP_ENDPOINT=https://api.honeycomb.io export OTEL_EXPORTER_OTLP_HEADERS=x-honeycomb-team=$HONEYCOMB_API_KEY export OTEL_TRACES_EXPORTER=otlp export OTEL_SERVICE_NAME=fusionauth-test export OTEL_TRACES_SAMPLER=traceidratio export OTEL_TRACES_SAMPLER_ARG=0.5 export OTEL_RESOURCE_ATTRIBUTES=SampleRate=2 export OTEL_INSTRUMENTATION_HTTP_CAPTURE_HEADERS_CLIENT_REQUEST=X-FusionAuth-TenantId
Start FusionAuth up and enjoy your new insights.
You can load test FusionAuth to determine if its performance will meet your calculated needs. The FusionAuth team tests the performance of FusionAuth regularly, but every situation is different and running your own load tests on your instance provides useful information.
You may use your own load testing tool. Any tool which can make HTTP requests, such as gatling or jmeter, will work.
There is an open source project which provides some useful scripts for load testing. This is a specialized library and primarily used for internal purposes, so use at your own risk. Scenarios currently supported include:
- Creating tenants
- Creating applications
- Registering users for applications
- Logging users in using the Login API
- A full Authorization Code grant
Performance bottlenecks are typically the database or the CPU nodes, depending on sizing of each of these components and primary functionality tested. For example, both logins and creating users use a lot of CPU due to password hashing. Make sure each of these components are monitored and sized appropriately.
If testing a multi-node system behind a load balancer, ensure that you have disabled sticky sessions, as they are not required for FusionAuth functionality.