Cannot login to FusionAuth dashboard

Description

Our FusionAuth instance was running with no errors until one day, when logging in to FusionAuth we are redirected to "Return to Login" page everytime.

Observed versions

• Version: 1.45.2

Expected behavior

Being able to log in to FusionAuth and get redirected to the Admin dashboard

Screenshots

Platform

• Device: Desktop
• OS: Windows, Apple
• Browser + version: Chrome Version 134.0.6998.37 (Official Build) (64-bit)
• Database PostgresSQL; Version: 9.6.18-apline
• FusionAuth server & FusionAuth DB are running withing a Kubernetes cluster

Attempted fixes:

• Restart FusionAuth
• Restart Database
• Tried multiple accounts

Additional context

• This problem seemed to occur due to an unexpected restart of the PostgresSQL database.
• The database is up & running.
• Debug log when the database restarted

2025-03-12 03:15:00.232 CET
### Error querying database. Cause: java.sql.SQLTransientConnectionException: HikariPool-1 - Connection is not available, request timed out after 2001ms.
2025-03-12 03:15:00.232 CET
### The error may exist in io/fusionauth/api/domain/InstanceMapper.java (best guess)
2025-03-12 03:15:00.232 CET
### The error may involve io.fusionauth.api.domain.InstanceMapper.retrieve
2025-03-12 03:15:00.232 CET
### The error occurred while executing a query
2025-03-12 03:15:00.232 CET
### Cause: java.sql.SQLTransientConnectionException: HikariPool-1 - Connection is not available, request timed out after 2001ms. at org.apache.ibatis.exceptions.ExceptionFactory.wrapException(ExceptionFactory.java:30) at org.apache.ibatis.session.defaults.DefaultSqlSession.selectList(DefaultSqlSession.java:153) at org.apache.ibatis.session.defaults.DefaultSqlSession.selectList(DefaultSqlSession.java:145) at org.apache.ibatis.session.defaults.DefaultSqlSession.selectList(DefaultSqlSession.java:140) at org.apache.ibatis.session.defaults.DefaultSqlSession.selectOne(DefaultSqlSession.java:76) at jdk.internal.reflect.GeneratedMethodAccessor6.invoke(Unknown Source) at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.base/java.lang.reflect.Method.invoke(Method.java:568) at org.apache.ibatis.session.SqlSessionManager$SqlSessionInterceptor.invoke(SqlSessionManager.java:357) at jdk.proxy2/jdk.proxy2.$Proxy52.selectOne(Unknown Source) at org.apache.ibatis.session.SqlSessionManager.selectOne(SqlSessionManager.java:166) at org.apache.ibatis.binding.MapperMethod.execute(MapperMethod.java:87) at org.apache.ibatis.binding.MapperProxy$PlainMethodInvoker.invoke(MapperProxy.java:145) at org.apache.ibatis.binding.MapperProxy.invoke(MapperProxy.java:86) at jdk.proxy2/jdk.proxy2.$Proxy59.retrieve(Unknown Source) at io.fusionauth.api.license.FusionAuthLicenseMetaDataManager.localLookup(FusionAuthLicenseMetaDataManager.java:50) at com.inversoft.license.v2.BaseConfigurationLicenseMetaDataManager.lookup(BaseConfigurationLicenseMetaDataManager.java:30) at com.inversoft.license.v2.local.DefaultLocalLicenseSystem.load(DefaultLocalLicenseSystem.java:35) at com.inversoft.license.v2.DefaultLicenseProvider$LocalReloader.run(DefaultLicenseProvider.java:131) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) at java.base/java.lang.Thread.run(Thread.java:833) Caused by: java.sql.SQLTransientConnectionException: HikariPool-1 - Connection is not available, request timed out after 2001ms. at com.zaxxer.hikari.pool.HikariPool.createTimeoutException(HikariPool.java:696) at com.zaxxer.hikari.pool.HikariPool.getConnection(HikariPool.java:181) at com.zaxxer.hikari.pool.HikariPool.getConnection(HikariPool.java:146) at com.zaxxer.hikari.HikariDataSource.getConnection(HikariDataSource.java:100) at org.apache.ibatis.transaction.jdbc.JdbcTransaction.openConnection(JdbcTransaction.java:139) at org.apache.ibatis.transaction.jdbc.JdbcTransaction.getConnection(JdbcTransaction.java:61) at org.apache.ibatis.executor.BaseExecutor.getConnection(BaseExecutor.java:337) at org.apache.ibatis.executor.SimpleExecutor.prepareStatement(SimpleExecutor.java:86) at org.apache.ibatis.executor.SimpleExecutor.doQuery(SimpleExecutor.java:62) at org.apache.ibatis.executor.BaseExecutor.queryFromDatabase(BaseExecutor.java:325) at org.apache.ibatis.executor.BaseExecutor.query(BaseExecutor.java:156) at org.apache.ibatis.executor.CachingExecutor.query(CachingExecutor.java:109) at org.apache.ibatis.executor.CachingExecutor.query(CachingExecutor.java:89) at org.apache.ibatis.session.defaults.DefaultSqlSession.selectList(DefaultSqlSession.java:151) ... 23 common frames omitted Caused by: org.postgresql.util.PSQLException: This connection has been closed. at org.postgresql.jdbc.PgConnection.checkClosed(PgConnection.java:907) at org.postgresql.jdbc.PgConnection.setNetworkTimeout(PgConnection.java:1649) at com.zaxxer.hikari.pool.PoolBase.setNetworkTimeout(PoolBase.java:561) at com.zaxxer.hikari.pool.PoolBase.isConnectionDead(PoolBase.java:168) at com.zaxxer.hikari.pool.HikariPool.getConnection(HikariPool.java:170) ... 35 common frames omitted
2025-03-12 03:15:01.330 CET
2025-03-12 02:15:01.329 AM ERROR io.fusionauth.api.service.system.DefaultAsyncTaskManager - An exception occurred while managing an async task.
2025-03-12 03:15:01.330 CET
org.apache.ibatis.exceptions.PersistenceException:
2025-03-12 03:15:01.330 CET
### Error querying database. Cause: java.sql.SQLTransientConnectionException: HikariPool-1 - Connection is not available, request timed out after 2000ms.
2025-03-12 03:15:01.330 CET
### The error may exist in io/fusionauth/api/domain/LockMapper.java (best guess)
2025-03-12 03:15:01.330 CET
### The error may involve io.fusionauth.api.domain.LockMapper.lock
2025-03-12 03:15:01.330 CET
### The error occurred while executing a query

• Debug log after restart

2025-03-12T09:14:09.786694464Z
{}
2025-03-12T09:14:09.786718924Z
2025-03-12 09:14:09.786 AM WARN com.inversoft.configuration.BasePropertiesFileInversoftConfiguration - Your FusionAuth configuration file [/usr/local/fusionauth/config/fusionauth.properties] needs attention. Here are the warnings:
2025-03-12T09:14:09.786772574Z
{}
2025-03-12T09:14:09.786782694Z
- You are using a deprecated configuration property name of [FUSIONAUTH_APP_HTTP_PORT]. The new allowed names for that property are [fusionauth-app.http.port]
2025-03-12T09:14:09.786789094Z
- You are using a deprecated configuration property name of [FUSIONAUTH_APP_HTTP_PORT]. The new allowed names for that property are [fusionauth-app.http.port]
2025-03-12T09:14:09.786794554Z
{}
2025-03-12T09:14:09.788385584Z
2025-03-12 09:14:09.788 AM INFO com.inversoft.jdbc.hikari.DataSourceProvider - Connecting to PostgreSQL database at [jdbc:postgresql://divizend-auth-db-0:5432/fusionauth]
2025-03-12T09:14:09.791886714Z
2025-03-12 09:14:09.791 AM WARN com.zaxxer.hikari.HikariConfig - HikariPool-1 - idleTimeout has been set but has no effect because the pool is operating as a fixed size pool.
2025-03-12T09:14:09.794193314Z
2025-03-12 09:14:09.793 AM INFO com.zaxxer.hikari.HikariDataSource - HikariPool-1 - Starting...
2025-03-12T09:14:09.819821924Z
2025-03-12 09:14:09.819 AM INFO com.zaxxer.hikari.pool.HikariPool - HikariPool-1 - Added connection org.postgresql.jdbc.PgConnection@6cbe68e9
2025-03-12T09:14:09.831727314Z
2025-03-12 09:14:09.831 AM INFO com.zaxxer.hikari.HikariDataSource - HikariPool-1 - Start completed.
2025-03-12T09:14:11.268084494Z
2025-03-12 09:14:11.267 AM INFO com.inversoft.scheduler.DefaultScheduler - Starting up scheduler
2025-03-12T09:14:11.268828044Z
2025-03-12 09:14:11.268 AM INFO com.inversoft.scheduler.DefaultScheduler - Scheduler is running
2025-03-12T09:14:11.439808283Z
2025-03-12 09:14:11.439 AM INFO io.fusionauth.api.service.system.NodeService - Node [deb1186a-32c4-493e-83a7-a5299b66f6a8] added with address [http://10.52.0.28:80]
2025-03-12T09:14:12.945987592Z
2025-03-12 09:14:12.944 AM ERROR com.inversoft.scheduler.LogAndRetainFailureHandler - The scheduled service [class io.fusionauth.api.service.cache.KeyCacheLoader] failed but will be re-run.
2025-03-12T09:14:12.946030462Z
io.fusionauth.pem.PEMDecoderException: java.security.InvalidKeyException: Could not decode the X.509 public key. Expected at 2 values in the DER encoded sequence but found [2] at io.fusionauth.pem.PEMDecoder.decode(PEMDecoder.java:126) at io.fusionauth.pem.domain.PEM.decode(PEM.java:111) at io.fusionauth.api.service.cache.KeyCacheLoader.internalLoad(KeyCacheLoader.java:101) at com.inversoft.cache.BaseCacheLoader.load(BaseCacheLoader.java:21) at io.fusionauth.api.service.cache.KeyCacheLoader.run(KeyCacheLoader.java:58) at com.inversoft.scheduler.InjectableRunnable.run(InjectableRunnable.java:56) at com.inversoft.scheduler.DefaultScheduler.<init>(DefaultScheduler.java:70) at com.inversoft.scheduler.DefaultScheduler$$FastClassByGuice$$208640461.GUICE$TRAMPOLINE(<generated>) at com.inversoft.scheduler.DefaultScheduler$$FastClassByGuice$$208640461.apply(<generated>) at com.google.inject.internal.DefaultConstructionProxyFactory$FastClassProxy.newInstance(DefaultConstructionProxyFactory.java:82) at com.google.inject.internal.ConstructorInjector.provision(ConstructorInjector.java:114) at com.google.inject.internal.ConstructorInjector.construct(ConstructorInjector.java:91) at com.google.inject.internal.ConstructorBindingImpl$Factory.get(ConstructorBindingImpl.java:300) at com.google.inject.internal.ProviderToInternalFactoryAdapter.get(ProviderToInternalFactoryAdapter.java:40) at com.google.inject.internal.SingletonScope$1.get(SingletonScope.java:169) at com.google.inject.internal.InternalFactoryToProviderAdapter.get(InternalFactoryToProviderAdapter.java:45) at com.google.inject.internal.FactoryProxy.get(FactoryProxy.java:60) at com.google.inject.internal.InternalInjectorCreator.loadEagerSingletons(InternalInjectorCreator.java:213) at com.google.inject.internal.InternalInjectorCreator.injectDynamically(InternalInjectorCreator.java:186) at com.google.inject.internal.InternalInjectorCreator.build(InternalInjectorCreator.java:113) at com.google.inject.Guice.createInjector(Guice.java:87) at com.google.inject.Guice.createInjector(Guice.java:69) at com.google.inject.Guice.createInjector(Guice.java:59) at org.primeframework.mvc.guice.GuiceBootstrap.initialize(GuiceBootstrap.java:60) at org.primeframework.mvc.BasePrimeMain.hup(BasePrimeMain.java:69) at org.primeframework.mvc.BasePrimeMain.start(BasePrimeMain.java:100) at io.fusionauth.app.FusionAuthMain.main(FusionAuthMain.java:27) Caused by: java.security.InvalidKeyException: Could not decode the X.509 public key. Expected at 2 values in the DER encoded sequence but found [2] at io.fusionauth.pem.PEMDecoder.decode_X_509(PEMDecoder.java:334) at io.fusionauth.pem.PEMDecoder.decode(PEMDecoder.java:112) ... 26 common frames omitted
2025-03-12T09:14:15.700435663Z
2025-03-12 09:14:15.699 AM INFO io.fusionauth.api.service.system.NodeService - Node [deb1186a-32c4-493e-83a7-a5299b66f6a8] promoted to master at [2025-03-12T09:14:15.699888323Z], the previous master Node [cf39ac34-f49e-4394-8721-8a2f8bfad22c] has been shutdown or removed
2025-03-12T09:14:16.467362998Z
2025-03-12 09:14:16.466 AM INFO io.fusionauth.app.primeframework.FusionHTTPContextAuthSetup - Initializing the FusionAuth HTTP Context.
2025-03-12T09:14:16.674891576Z
2025-03-12 09:14:16.674 AM INFO org.primeframework.mvc.PrimeMVCRequestHandler - Initializing Prime
2025-03-12T09:14:16.676199576Z
2025-03-12 09:14:16.675 AM INFO org.primeframework.mvc.PrimeMVCRequestHandler - Initializing Prime
2025-03-12T09:14:16.676563276Z
2025-03-12 09:14:16.676 AM INFO io.fusionauth.http.server.HTTPServer - Starting the HTTP server. Buckle up!
2025-03-12T09:14:16.684964236Z
2025-03-12 09:14:16.684 AM INFO io.fusionauth.http.server.HTTPServer - HTTP server listening on port [80]
2025-03-12T09:14:16.684994656Z
2025-03-12 09:14:16.684 AM INFO io.fusionauth.http.server.HTTPServer - HTTP server started successfully
2025-03-12T09:14:16.685016076Z
2025-03-12 09:14:16.684 AM INFO io.fusionauth.http.server.HTTPServer - Starting the HTTP server. Buckle up!
2025-03-12T09:14:16.685700126Z
2025-03-12 09:14:16.685 AM INFO io.fusionauth.http.server.HTTPServer - HTTP server listening on port [9012]
2025-03-12T09:14:16.685722766Z
2025-03-12 09:14:16.685 AM INFO io.fusionauth.http.server.HTTPServer - HTTP server started successfully