Troubleshooting

System Log UI

The system logs may be reviewed in the FusionAuth admin UI by navigating to System → Logs. This feature was added in version 1.16.0. This UI will only render logs for systems that write their logs out to disk. Deployments that write logs to STDOUT, such is the case with Docker, will not be able to review logs through this interface.

The system log UI organizes and renders the most recent 64KB of all logs for all nodes in the deployment. If you need the entire log, use the download action in the upper-right hand corner of UI to download a zip of all logs.

Filesystem Logs

Alternatively, the logs may be accessed directly. The following are the default locations for each of the FusionAuth log files. You may or may not have all of the FusionAuth services installed for your system, so you may not have all of the following logs on your server.

/usr/local/fusionauth/logs/fusionauth-app.log
/usr/local/fusionauth/logs/fusionauth-search.log

These paths assume the suggested product location of \fusionauth. This path may be different on your system depending on where you unpacked the zip files.

\fusionauth\logs\fusionauth-app.log
\fusionauth\logs\fusionauth-search.log

Note that if you started Windows via Fast Path, the fusionauth-app.log file will not be created. Instead, the services are running interactively and all logging is written to stdout.

Event Log

The event log is a FusionAuth message store designed to capture different types of events that may be necessary to communicate to a FusionAuth developer or admin user.

The event log may contain helpful details to indicate the cause of the failure, or a failure condition you need to be aware of in FusionAuth. See System → Event Log to view this log.

While not limited to, generally speaking, the event log will contain events or errors related to external systems or asynchronous issues that are difficult to communicate to the API caller or the FusionAuth admin at runtime. While not intended to be an exhaustive list, examples of these types of errors are:

• SMTP connection issues

• Lambda invocation errors

• External identity provider failures or configuration issues

• Failure to deliver a webhook event

Functionality specific

Many features have additional debugging that can be enabled via the administrative user interface or the API. These features include, but are not limited to:

• Identity Providers

• Application SAML IdP configuration

• Connectors

If you are having issues with certain functionality, look for a Debug enabled checkbox in the user interface. If present, set it to true.

The additional debugging information will be written to the Event Log.

Enabling JMX

JMX is an API that allows deeper insight and monitoring of Java applications, including FusionAuth. To use JMX, you’ll need to include a few modules:

• jdk.management.agent

• jdk.httpserver if you are using the JMX http adapter

These should be part of a recent, standard Java install. However, if you are using the Docker image, which is built with jlink, you’ll need to build a new image with these modules. There’s a GitHub issue about adding these modules to the FusionAuth image.

When the above modules are available to your Java runtime, configure JMX by passing additional arguments to FusionAuth using any of the supported configuration options such as adding a fusionauth-app.additional-java-args entry to the fusionauth.properties file.

Send A Test Email

The first step is to ensure that you can send a test email. Navigate to Tenants → Your Tenant → Edit → Email and send a test email. If it is received, FusionAuth can send emails via SMTP.

If sending a test email fails, a tool such as SWAKS can help debug SMTP issues. Using this tool removes FusionAuth from the equation.

Further Investigation

If the test email succeeds, but you aren’t receiving other emails, investigate further by following these steps:

• Send emails to different addresses at different providers and check the spam folder.

• Make sure that you are firing the event that you expect to send email correctly. For instance, if you are looking for email verification of users, make sure that is enabled.

• Test with a different FusionAuth function. Sending a password reset email is easy to do from the user details screen.

• Make sure the email template is correct, or use a default template.

Additionally, confirm that the template content matches up with the template type. Otherwise template variables will be missing. If this occurs, you may see a message like this in the Event Log.

Render Errors
---------------
html : freemarker.core.InvalidReferenceException: The following has evaluated to null or missing:
==> changePasswordId  [in nameless template at line 5, column 59]

SMTP Logging

If you would like to see verbose SMTP logging, follow these steps:

Doing this logs the full SMTP conversation, which can be verbose. You should remove this setting when you have finished troubleshooting.

SAML and Apple Identity Providers

SAML and Apple Identity Providers will not work without the proper CORS configuration. If CORS is misconfigured, you will see a a 403 status code during the login process in the browser network console.

When setting up a SAML Identity Provider in FusionAuth, ensure you have configured CORS correctly.

When setting up an Apple Identity Provider in FusionAuth, ensure you have configured CORS correctly.

To further debug CORS issues:

• Enable debugging by navigating to Settings → System → CORS, selecting Debug enabled and saving the settings.

• Attempt to log in again using the provider.

• Review the Event Log for relevant messages. See System → Event Log to view this log.

After debugging is complete, ensure you disable CORS debugging to avoid spamming the Event Log with unnecessary information. Disable debugging by navigating to Settings → System → CORS, deselecting Debug enabled and saving the settings.

Theme Errors Prevent Login

If you have edited a template and it is causing errors preventing you from logging in to FusionAuth, you can override the use of the UI templates. This will render a form allowing you to log in. To do this:

• Open your browser and access your FusionAuth admin UI.

• This will redirect you to the broken /oauth2/authorize page.

• Click in your browser’s address bar and scroll to the end.

• Add the String &bypassTheme=true to the end of the URL and hit the Enter key.

This should render the default login page that ships with FusionAuth and allow you to log in and fix any errors you have.

Default Theme Used Incorrectly

Anytime a request is made to a themed page and we are unable to identify the tenant, the default tenant will be used. This includes, but is not limited to:

• The root page / when a client_id or tenantId is not provided.

• Any themed pages such as /password/forgot when a client_id is not provided.

• Edge case error conditions where FusionAuth doesn’t have context to determine the application or tenant.

If you see the default theme unexpectedly, ensure you are passing required parameters, such as the tenantId or client_id, to the page so that it can determine the applicable application or tenant. These parameters allow FusionAuth to determine the correct theme to display.

Not Registered

Please note, if you are not registered for an application, you will not be able to manage your account information from these self service account management.

From the Admin UI click on Users → Manage → Registrations Tab → Add registration → Add User. Another option is to be programmatically added to the application through an API call.

Registered But Configuration Needed

If you see this, you have to select a form to use.

To set a self service form, click under Applications→ Edit Application → Registration

Select the form you would like to use. A default form is shipped with FusionAuth.

A Final Option

Please review the procedure outlined below if attempting set up a clean install or upgrade using RPM’s. However, this is an option of final recourse and only necessary if unexpected upgrade or installation issues arise.

The following steps will produce a clean installation preserving:

• The FusionAuth config file found in /usr/local/fusionauth/config/fusionauth.properties

• The Elasticsearch index found in /usr/local/fusionauth/data

Below assumes both FusionAuth App and FusionAuth Search will be uninstalled and reinstalled together (your setup may vary).

Access Denied

If you see an Access Denied error in your browser while using the FusionAuth UI, this could be caused by a misconfigured CDN or proxy. This error often produces a 4xx error code such as a 403. To fix this issue, ensure that your CDN, proxy, or gateway does not prevent traffic from flowing directly to FusionAuth. FusionAuth can handle all HTTP traffic and any network handling between the browser and FusionAuth should be as simple as possible.

FusionAuth Stops Unexpectedly

If you are running FusionAuth on a server with limited memory, it may stop unexpectedly. This can also happen when many other applications are competing for memory on the server. Make sure you verify you are running with the minimum requirements specified in the system requirements.

When FusionAuth is killed because of an out-of-memory issue (OOM), it dies with no explanation in the FusionAuth logs. You might see lines like this in server system logs (typically under /var/log/):

Dec 30 12:00:38 vps kernel: Out of memory: Kill process 30047 (java) score 98 or sacrifice child

The OOM killer will begin killing services once the kernel runs out of memory. The solution will be to allocate less memory to FusionAuth or to increase the amount of RAM available.

You can do the former with the fusionauth-app.memory setting. See the configuration reference for more details.

You may do the latter by running a larger server or running fewer competing applications. In particular, applications used by FusionAuth such as Elasticsearch or the database may be moved off to external servers.

fusionauth_1 exited with code 137

MapperParsingException

If you are using the Elasticsearch search engine, attributes in the user.data and registration.data are indexed for you. The data fields can contain arbitrary JSON objects.

If your user search is failing mysteriously and you see an error like this one in your system logs:

org.elasticsearch.index.mapper.MapperParsingException: failed to parse field [data.field] of type [text] in document with id 'b8f1ad7d-def0-48d1-a983-aeabf0122b90'. Preview of field's value: '{bar=123}'

or

org.elasticsearch.index.mapper.MapperParsingException: Could not dynamically add mapping for field [field.email]. Existing mapping for [data.field] must be of type object but found [text].

This means that the type of a field in one of the data fields was changed. In this case, data.field was originally text, but the request causing this error message was trying to put an object into that attribute.

Elasticsearch will dynamically build a schema based upon custom data, so it is critical that the schema not be modified.

The fix is:

• Understand how the field types changed.

• Find the user’s Id.

• Modify the problematic data fields via the User API to remove the mismatched fields.

Unfortunately, you can’t search for the user by email, username, or other characteristics using the User Search API. If you have the user’s Id, which may be available in the error messages, you can use that. Searching by UUID doesn’t use Elasticsearch. If you do not have the UUID, you should change to the database search engine to find the user. You can switch back to Elasticsearch after you have corrected the data field.

If you have a support contract, please open a ticket and we’ll help you out. If not, please back up your FusionAuth database and test the fix thoroughly in a staging environment before applying it to your production instance.