Troubleshooting

Overview

If any problems arise or if you are unable to access FusionAuth, consult the FusionAuth log files. In most cases API errors will be written out to the fusionauth-app.log file, but for some installations such as Docker, the errors will be sent to stdout.

If you need further assistance, please ask a question in the FusionAuth forum or open an issue on Github if you have found a bug (please provide replication steps).

If you have a paid edition which includes support you may open a support request from your FusionAuth Account page. Learn more about purchasing support.

Here’s more information about getting technical support.

• Logs

• Enabling Debugging

• Troubleshooting Email

• Troubleshooting Themes

• Troubleshooting Two Factor Authentication

• Troubleshooting Database Connections

• Troubleshooting Performance

• Troubleshooting Kickstart

• Troubleshooting Self Service Account Management

• Common Errors

Logs

System Log UI

The system logs may be reviewed in the FusionAuth admin UI by navigating to System → Logs. This feature was added in version 1.16.0. This UI will only render logs for systems that write their logs out to disk. Deployments which write logs to STDOUT, such is the case with Docker, will not be able to review logs through this interface.

The system log UI organizes and renders the most recent 64KB of all logs for all nodes in the deployment. If you need the entire log, use the download action in the upper-right hand corner of UI to download a zip of all logs.

/usr/local/fusionauth/logs/fusionauth-app.log
/usr/local/fusionauth/logs/fusionauth-search.log

\fusionauth\logs\fusionauth-app.log
\fusionauth\logs\fusionauth-search.log

Enabling Debugging

Functionality specific

Many features have additional debugging that can be enabled via the administrative user interface or the API. These features include, but are not limited to:

• Identity Providers

• Application SAML IdP configuration

• Connectors

If you are having issues with certain functionality, look for a Debug enabled checkbox in the user interface. If present, set it to true.

The additional debugging information will be written to the Event Log.

Troubleshooting Email

FusionAuth sends a lot of email, for forgotten passwords, passwordless login and other features.

Troubleshooting email delivery is difficult. There are many factors affecting it. However, there are steps you can take to narrow down the problem.

Send A Test Email

The Send Test Email button has been available since 1.16.0

The first step is to ensure that you can send a test email. Navigate to Tenants → Your Tenant → Edit → Email and send a test email. If it is received, FusionAuth can send emails via SMTP.

If sending a test email fails, a tool such as SWAKS can help debug SMTP issues. Using this tool removes FusionAuth from the equation.

Troubleshooting SAML or OIDC Identity Provider Configuration

A common pattern is to set up SAML or OIDC as the login method for the FusionAuth application. This lets you keep all your users in the SAML or OIDC system, and have FusionAuth delegate auth decisions to it. However, if your Identity Provider isn’t configured correctly, you can end up in a dead end, unable to log in. This might happen if you:

• Set up SAML or OIDC as the authentication method for your FusionAuth admin application.

• Set the Managed domains to your domain. For example, example.com.

• Don’t have any users with a domain that is different than example.com.

• Accidentally misconfigure SAML or OIDC and break authentication.

• Log out of the FusionAuth admin application.

You won’t be able to log in to the FusionAuth admin application to correct the misconfiguration. You’ve locked down the FusionAuth admin application so that only valid logins can access it, but because of the misconfiguration, there is no account with a valid SAML or OIDC login.

This is a problem, but not an insurmountable one. Your options depend on when you discover the issue. If you are beginning your SAML or OIDC configuration, you can avoid this scenario. Follow these steps:

• Don’t set any value of Managed domains before you have tested the SAML or OIDC configuration.

• Test authentication in a different or incognito browser window, ensuring that an admin user account is always accessible to modify configuration.

• Add an admin user account with a domain not in the Managed domains setting. Ensure this user is registered with the FusionAuth admin application and has the admin role.

• Set up an API key. Navigate to Settings → API Keys to do so and make sure it has privileges to create users and registrations. This will open up options in the future to reset settings without using the administrative user interface.

If you are currently locked out of your FusionAuth application, you have fewer options:

• If you have a known username and password that are not delegated to SAML or OIDC (perhaps the initial account created when you set up FusionAuth) proceed to the FusionAuth admin login page. Append &showPasswordField=true to the end of the login URL. This will force the UI to show the password field.

• If you have an API key with appropriate privileges, you can modify the configuration without using the administrative user interface. Add an admin user with a different email domain and sign into the admin interface to correct your SAML or OIDC configuration. Here’s how you might do so (3c219e58-ed0e-4b18-ad48-f4f92793ae32 is the FusionAuth application Id):

curl -XPOST -H "Authorization: $API_KEY" -H "Content-Type: application/json" \
-d '{"user": {"email": "user@differentdomain.com","password":"password"}, "registration": {"applicationId" : "3c219e58-ed0e-4b18-ad48-f4f92793ae32","roles":["admin"]}}' \
https://local.fusionauth.io/api/user/registration

If you have neither an API key nor a known user, you can restore from a database backup, modify the database directly if you have access, or re-install FusionAuth.

Troubleshooting Themes

If you happen to get into a situation where you have edited a template and it is causing errors that are preventing you from logging in, you can override the use of the UI templates to render a login form that lets you log in.

To do this, open your browser and access your FusionAuth admin UI. This will redirect you to the broken /oauth2/authorize page. Click in your browsers address bar and scroll to the end. Finally, add the String &bypassTheme=true to the end of the URL and hit the Enter key. This should render the default login page that ships with FusionAuth and allow you to log in and fix any errors you have.

Troubleshooting Two Factor Authentication

If you are receiving an invalid code for your two factor authentication and you are using a time based one time password (TOTP) application such as Google Authenticator or Authy, confirm that the system time is correct on the server.

The TOTP two factor auth system is time dependent and if there is any slippage between the system time of the client and the system time of the server, the generated code will not be correct. This is also known as "clock skew".

The fix is to ensure that the FusionAuth server has the correct system time. How exactly do that that depends on the type of system; please consult your operating system documentation.

Troubleshooting Database Connections

If you find that FusionAuth is unable to connect to the database, ensure you are able to use a command line client connection to successfully make a connection using the same port and credentials.

Some MySQL services such as Microsoft Azure may require a specific version of TLS to connect successfully. At the time of writing this note, the MySQL connector will not attempt to utilize TLSv1.2 by default, so when connecting to a service that requires this version you will need to explicitly request this version of TLS on the connection string. For example, appending this enabledTLSProtocols=TLSv1.2 to the connection string will allow you to successfully connect to an Azure managed MySQL database. See MySQL Connector : Connecting Securely Using SSL for additional information.

Troubleshooting Performance

If you are not seeing the logins per second you’d expect from FusionAuth, you can take some steps to troubleshoot the issue. Generally speaking the primary bottleneck for logins per second is CPU. Hashing the password is intentionally slow. FusionAuth will not be able to perform more logins per second than your CPU can handle. The database is the other likely bottleneck.

One way to identify if the password hashing is the bottleneck in load tests is to reduce the hash strength. See Tenants → Edit → Password → Cryptographic hash settings. Set this to Salted MD5 with a factor of 1 and then enable Re-hash on login. This will cause each user to have their password re-hashed next time they login to use MD5. Salted MD5 is not a good password hashing strategy for a number of reasons, but is useful to test whether your system is CPU bound. Make sure you switch back after testing is done.

If you can still get a lower number of logins per second than you’d expect with this configuration, then the database is likely the bottleneck and you should update your system to use a database with more resources and load test that system.

If the MD5 configuration allows you to achieve a much higher logins per second, then the CPU is your bottleneck. If you are CPU bound, the only way to get more logins per second is to horizontally scale (add more nodes) or vertically scale (use larger CPUs with each node).

Troubleshooting Kickstart

If your Kickstart file doesn’t run on your instance, here are some troubleshooting steps to take.

• Ensure you have provided a Kickstart JSON file via the appropriate configuration parameter.

• Make sure the file is valid JSON, using a linter.

• If the file references any other files (such as email templates), ensure they exist as well.

• Kickstart will not run if it sees any users, API keys, or applications in the FusionAuth database. This is to prevent data loss. If you can log in to the FusionAuth administrative user interface, Kickstart will not run.

Additionally, Kickstart takes time to run, depending on how many API calls you make and what kind of API calls they are. If you are starting up an instance with Kickstart and you visit the FusionAuth administrative user interface in your browser, you may see the Setup Wizard briefly while Kickstart is running. After Kickstart finishes, you should not see the Setup Wizard. This may require a browser refresh.

Troubleshooting Self Service Account Management

Not able to get your account management page up and running? Below are two common warnings and how to fix them.

• Not Registered

• Registered But Configuration Needed

Not Registered

Please note, if you are not registered for an application, you will not be able to manage your account information from these self service account management.

From the Admin UI click on Users → Manage → Registrations Tab → Add registration → Add User. Another option is to be programmatically added to the application through an API call.

Registered But Configuration Needed

If you see this, you have to select a form to use.

To set a self service form, click under Applications→ Edit Application → Registration

Select the form you would like to use. A default form is shipped with FusionAuth.

Common Errors

Dec 30 12:00:38 vps kernel: Out of memory: Kill process 30047 (java) score 98 or sacrifice child

org.elasticsearch.index.mapper.MapperParsingException: failed to parse field [data.field] of type [text] in document with id 'b8f1ad7d-def0-48d1-a983-aeabf0122b90'. Preview of field's value: '{bar=123}'

org.elasticsearch.index.mapper.MapperParsingException: Could not dynamically add mapping for field [field.email]. Existing mapping for [data.field] must be of type object but found [text].