Monitoring FusionAuth

Overview

Once you’ve installed FusionAuth and integrated it with your applications, you have to run it. For that you need insight into how the system is performing. There are a number of ways to gain this insight.

Monitoring Summary Table

Here’s an overview table of monitoring options.

Monitoring Overview

NameContainsIngestion OptionsMore Information
System LogsExceptions, stack traces, database connection issues, Elasticsearch connection issues. Minimal tracing capability unless debug logging is enabled.Can be exported as a zip file via API.More info
Audit LogsCreated by the admin user interface actions. This is just an API, so a customer can also call this API. This is a free form API, and the audit log contents are whatever you put in.This can be consumed via web hook or API.More info
Event LogsDebug information for external integrations with IdPs, SMTP etc. In general, runtime errors that are typically not caused by FusionAuth and cannot be communicated well at runtime. Examples: a template rendering error in a custom theme, an exception connecting to SMTP due to bad credentials, a failure in a SAML exchange, a connection to a webhook is failing.This can be consumed via web hook or API.More info
Login recordsEach successful login is recorded here. Includes IP information, application, user, and timestamp.Can be consumed via API. This record itself is not sent through a webhook, but a login success or login failure can be consumed via a web hook.More info
WebhooksTriggered by events as documented, sending is configurable. Contains IP and location information when available.Can be sent to an HTTP endpoint or a Kafka topic.More info
MetricsSystem metrics, such as JVM buffers, memory and threads.Can be consumed via API and available in Prometheus compatible format.More info

Log Files

The system log files will be placed in the logs directory under the FusionAuth installation unless you are running FusionAuth in a container. In the latter case, the log output will be sent to STDOUT by default. You may also set up a Docker logging driver to direct log files elsewhere.

System logs running in non-containerized instances can also be exported via the Export System Logs API.

Learn more about FusionAuth log files.

Application Logging

There are a few different APIs exposing FusionAuth application specific information you may want to ingest into your monitoring system:

In general these are APIs you will have to poll to ingest.

Log File Formats

Log File Information

Log TypeExport FormatTimezoneDate FormatAPI Docs
Audit LogsZipped CSVSpecified by the zoneId parameterSpecified by the dateTimeSecondsFormat parameter, defaults to M/d/yyyy hh:mm:ss a zAPI
Event LogsJSONUTCInstantAPI
Login RecordsZipped CSVSpecified by the zoneId parameterSpecified by the dateTimeSecondsFormat parameter, defaults to M/d/yyyy hh:mm:ss a zAPI
System LogsZipped directory of files. Log entries separated by newlines, but may be unstructured (stack traces, etc).For log entries, the timezone of the server. The zoneId parameter, if provided, is used to build the filename.yyyy-MM-dd h:mm:ss.SSS aAPI

There are currently no plugins for ingesting FusionAuth logs into a log management system. A polling script using a client library is usually sufficient. Please file an issue if this does not meet your use case.

Application Events

There are a number of events, such as a failed authentication or an account deletion, that you may want to ingest into your monitoring system. These are available as webhooks.

Here’s the list of all available events.

Metrics

You can pull system metrics from the System API. The format of these metrics are evolving and thus are not documented.

You can also enable JMX metrics as outlined in the Troubleshooting documentation.

Prometheus Endpoint

The default system metrics are also available in a Prometheus compatible form. This tutorial explains how to set up Prometheus to scrape the FusionAuth metrics.

OpenTelemetry

FusionAuth is a Java application. You can use the OpenTelemetry project to instrument it. Doing so can assist troubleshooting performance issues and make it easier to run.

FusionAuth has no custom tracing spans embedded, but it can be set up as an OpenTelemetry exporter. Doing so provides useful data about types of requests, database queries, and more.

Exploring telemetry using Honeycomb.

Set Up the Exporter

Download the Java OpenTelemetry agent. Make it available to the FusionAuth instance. If you are using a zip/deb/rpm, put it in a directory. If using docker, mount the host directory to the container. Set an environment variable OT_PATH to point to the directory.

Next, you need to configure FusionAuth to use the agent by passing the correct Java command line switches. There are a couple of ways to do this.

You can use Java system properties, environment variables, or edit the fusionauth.properties config file. See the configuration reference for more info. For this example, environment variables will be used.

Set this environment variable and make sure it is available when you start FusionAuth (in a startup script, .bashrc, etc):

Enabling OpenTelemetry export

export FUSIONAUTH_APP_ADDITIONAL_JAVA_ARGS="-javaagent:$OT_PATH/opentelemetry-javaagent.jar"

Next, configure OpenTelemetry to point to your collector. Here’s an example of the environment variables to set to use HoneyComb as your collector. The specific variables needed will vary based on your chosen solution; please consult the collector’s documentation.

Configuring OpenTelemetry collector

# from https://docs.honeycomb.io/getting-data-in/opentelemetry/java-distro/#using-opentelemetry-without-the-honeycomb-distribution
export OTEL_EXPORTER_OTLP_ENDPOINT=https://api.honeycomb.io
export OTEL_EXPORTER_OTLP_HEADERS=x-honeycomb-team=$HONEYCOMB_API_KEY
export OTEL_TRACES_EXPORTER=otlp
export OTEL_SERVICE_NAME=fusionauth-test
export OTEL_TRACES_SAMPLER=traceidratio
export OTEL_TRACES_SAMPLER_ARG=0.5
export OTEL_RESOURCE_ATTRIBUTES=SampleRate=2
export OTEL_INSTRUMENTATION_HTTP_CAPTURE_HEADERS_CLIENT_REQUEST=X-FusionAuth-TenantId

Start FusionAuth up and enjoy your new insights.

Load Testing

You can load test FusionAuth to determine if its performance will meet your calculated needs. The FusionAuth team tests the performance of FusionAuth regularly, but every situation is different and running your own load tests on your instance provides useful information.

You may use your own load testing tool. Any tool which can make HTTP requests, such as gatling or jmeter, will work.

There is an open source project which provides some useful scripts for load testing. This is a specialized library and primarily used for internal purposes, so use at your own risk. Scenarios currently supported include:

  • Creating tenants
  • Creating applications
  • Registering users for applications
  • Logging users in using the Login API
  • A full Authorization Code grant

Tips

Performance bottlenecks are typically the database or the CPU nodes, depending on sizing of each of these components and primary functionality tested. For example, both logins and creating users use a lot of CPU due to password hashing. Make sure each of these components are monitored and sized appropriately.

If testing a multi-node system behind a load balancer, ensure that you have disabled sticky sessions, as they are not required for FusionAuth functionality.