Description

Our FusionAuth instance was running with no errors until one day, when logging in to FusionAuth we are redirected to "Return to Login" page everytime.

Observed versions

• Version: 1.45.2

Expected behavior

Being able to log in to FusionAuth and get redirected to the Admin dashboard

Screenshots

Platform

• Device: Desktop
• OS: Windows, Apple
• Browser + version: Chrome Version 134.0.6998.37 (Official Build) (64-bit)
• Database PostgresSQL; Version: 9.6.18-apline
• FusionAuth server & FusionAuth DB are running withing a Kubernetes cluster

Attempted fixes:

• Restart FusionAuth
• Restart Database
• Tried multiple accounts

Additional context

• This problem seemed to occur due to an unexpected restart of the PostgresSQL database.
• The database is up & running.
• Debug log when the database restarted

2025-03-12 03:15:00.232 CET
### Error querying database. Cause: java.sql.SQLTransientConnectionException: HikariPool-1 - Connection is not available, request timed out after 2001ms.
2025-03-12 03:15:00.232 CET
### The error may exist in io/fusionauth/api/domain/InstanceMapper.java (best guess)
2025-03-12 03:15:00.232 CET
### The error may involve io.fusionauth.api.domain.InstanceMapper.retrieve
2025-03-12 03:15:00.232 CET
### The error occurred while executing a query
2025-03-12 03:15:00.232 CET
### Cause: java.sql.SQLTransientConnectionException: HikariPool-1 - Connection is not available, request timed out after 2001ms. at org.apache.ibatis.exceptions.ExceptionFactory.wrapException(ExceptionFactory.java:30) at org.apache.ibatis.session.defaults.DefaultSqlSession.selectList(DefaultSqlSession.java:153) at org.apache.ibatis.session.defaults.DefaultSqlSession.selectList(DefaultSqlSession.java:145) at org.apache.ibatis.session.defaults.DefaultSqlSession.selectList(DefaultSqlSession.java:140) at org.apache.ibatis.session.defaults.DefaultSqlSession.selectOne(DefaultSqlSession.java:76) at jdk.internal.reflect.GeneratedMethodAccessor6.invoke(Unknown Source) at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.base/java.lang.reflect.Method.invoke(Method.java:568) at org.apache.ibatis.session.SqlSessionManager$SqlSessionInterceptor.invoke(SqlSessionManager.java:357) at jdk.proxy2/jdk.proxy2.$Proxy52.selectOne(Unknown Source) at org.apache.ibatis.session.SqlSessionManager.selectOne(SqlSessionManager.java:166) at org.apache.ibatis.binding.MapperMethod.execute(MapperMethod.java:87) at org.apache.ibatis.binding.MapperProxy$PlainMethodInvoker.invoke(MapperProxy.java:145) at org.apache.ibatis.binding.MapperProxy.invoke(MapperProxy.java:86) at jdk.proxy2/jdk.proxy2.$Proxy59.retrieve(Unknown Source) at io.fusionauth.api.license.FusionAuthLicenseMetaDataManager.localLookup(FusionAuthLicenseMetaDataManager.java:50) at com.inversoft.license.v2.BaseConfigurationLicenseMetaDataManager.lookup(BaseConfigurationLicenseMetaDataManager.java:30) at com.inversoft.license.v2.local.DefaultLocalLicenseSystem.load(DefaultLocalLicenseSystem.java:35) at com.inversoft.license.v2.DefaultLicenseProvider$LocalReloader.run(DefaultLicenseProvider.java:131) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) at java.base/java.lang.Thread.run(Thread.java:833) Caused by: java.sql.SQLTransientConnectionException: HikariPool-1 - Connection is not available, request timed out after 2001ms. at com.zaxxer.hikari.pool.HikariPool.createTimeoutException(HikariPool.java:696) at com.zaxxer.hikari.pool.HikariPool.getConnection(HikariPool.java:181) at com.zaxxer.hikari.pool.HikariPool.getConnection(HikariPool.java:146) at com.zaxxer.hikari.HikariDataSource.getConnection(HikariDataSource.java:100) at org.apache.ibatis.transaction.jdbc.JdbcTransaction.openConnection(JdbcTransaction.java:139) at org.apache.ibatis.transaction.jdbc.JdbcTransaction.getConnection(JdbcTransaction.java:61) at org.apache.ibatis.executor.BaseExecutor.getConnection(BaseExecutor.java:337) at org.apache.ibatis.executor.SimpleExecutor.prepareStatement(SimpleExecutor.java:86) at org.apache.ibatis.executor.SimpleExecutor.doQuery(SimpleExecutor.java:62) at org.apache.ibatis.executor.BaseExecutor.queryFromDatabase(BaseExecutor.java:325) at org.apache.ibatis.executor.BaseExecutor.query(BaseExecutor.java:156) at org.apache.ibatis.executor.CachingExecutor.query(CachingExecutor.java:109) at org.apache.ibatis.executor.CachingExecutor.query(CachingExecutor.java:89) at org.apache.ibatis.session.defaults.DefaultSqlSession.selectList(DefaultSqlSession.java:151) ... 23 common frames omitted Caused by: org.postgresql.util.PSQLException: This connection has been closed. at org.postgresql.jdbc.PgConnection.checkClosed(PgConnection.java:907) at org.postgresql.jdbc.PgConnection.setNetworkTimeout(PgConnection.java:1649) at com.zaxxer.hikari.pool.PoolBase.setNetworkTimeout(PoolBase.java:561) at com.zaxxer.hikari.pool.PoolBase.isConnectionDead(PoolBase.java:168) at com.zaxxer.hikari.pool.HikariPool.getConnection(HikariPool.java:170) ... 35 common frames omitted
2025-03-12 03:15:01.330 CET
2025-03-12 02:15:01.329 AM ERROR io.fusionauth.api.service.system.DefaultAsyncTaskManager - An exception occurred while managing an async task.
2025-03-12 03:15:01.330 CET
org.apache.ibatis.exceptions.PersistenceException:
2025-03-12 03:15:01.330 CET
### Error querying database. Cause: java.sql.SQLTransientConnectionException: HikariPool-1 - Connection is not available, request timed out after 2000ms.
2025-03-12 03:15:01.330 CET
### The error may exist in io/fusionauth/api/domain/LockMapper.java (best guess)
2025-03-12 03:15:01.330 CET
### The error may involve io.fusionauth.api.domain.LockMapper.lock
2025-03-12 03:15:01.330 CET
### The error occurred while executing a query

• Debug log after restart

2025-03-12T09:14:09.786694464Z
{}
2025-03-12T09:14:09.786718924Z
2025-03-12 09:14:09.786 AM WARN com.inversoft.configuration.BasePropertiesFileInversoftConfiguration - Your FusionAuth configuration file [/usr/local/fusionauth/config/fusionauth.properties] needs attention. Here are the warnings:
2025-03-12T09:14:09.786772574Z
{}
2025-03-12T09:14:09.786782694Z
- You are using a deprecated configuration property name of [FUSIONAUTH_APP_HTTP_PORT]. The new allowed names for that property are [fusionauth-app.http.port]
2025-03-12T09:14:09.786789094Z
- You are using a deprecated configuration property name of [FUSIONAUTH_APP_HTTP_PORT]. The new allowed names for that property are [fusionauth-app.http.port]
2025-03-12T09:14:09.786794554Z
{}
2025-03-12T09:14:09.788385584Z
2025-03-12 09:14:09.788 AM INFO com.inversoft.jdbc.hikari.DataSourceProvider - Connecting to PostgreSQL database at [jdbc:postgresql://divizend-auth-db-0:5432/fusionauth]
2025-03-12T09:14:09.791886714Z
2025-03-12 09:14:09.791 AM WARN com.zaxxer.hikari.HikariConfig - HikariPool-1 - idleTimeout has been set but has no effect because the pool is operating as a fixed size pool.
2025-03-12T09:14:09.794193314Z
2025-03-12 09:14:09.793 AM INFO com.zaxxer.hikari.HikariDataSource - HikariPool-1 - Starting...
2025-03-12T09:14:09.819821924Z
2025-03-12 09:14:09.819 AM INFO com.zaxxer.hikari.pool.HikariPool - HikariPool-1 - Added connection org.postgresql.jdbc.PgConnection@6cbe68e9
2025-03-12T09:14:09.831727314Z
2025-03-12 09:14:09.831 AM INFO com.zaxxer.hikari.HikariDataSource - HikariPool-1 - Start completed.
2025-03-12T09:14:11.268084494Z
2025-03-12 09:14:11.267 AM INFO com.inversoft.scheduler.DefaultScheduler - Starting up scheduler
2025-03-12T09:14:11.268828044Z
2025-03-12 09:14:11.268 AM INFO com.inversoft.scheduler.DefaultScheduler - Scheduler is running
2025-03-12T09:14:11.439808283Z
2025-03-12 09:14:11.439 AM INFO io.fusionauth.api.service.system.NodeService - Node [deb1186a-32c4-493e-83a7-a5299b66f6a8] added with address [http://10.52.0.28:80]
2025-03-12T09:14:12.945987592Z
2025-03-12 09:14:12.944 AM ERROR com.inversoft.scheduler.LogAndRetainFailureHandler - The scheduled service [class io.fusionauth.api.service.cache.KeyCacheLoader] failed but will be re-run.
2025-03-12T09:14:12.946030462Z
io.fusionauth.pem.PEMDecoderException: java.security.InvalidKeyException: Could not decode the X.509 public key. Expected at 2 values in the DER encoded sequence but found [2] at io.fusionauth.pem.PEMDecoder.decode(PEMDecoder.java:126) at io.fusionauth.pem.domain.PEM.decode(PEM.java:111) at io.fusionauth.api.service.cache.KeyCacheLoader.internalLoad(KeyCacheLoader.java:101) at com.inversoft.cache.BaseCacheLoader.load(BaseCacheLoader.java:21) at io.fusionauth.api.service.cache.KeyCacheLoader.run(KeyCacheLoader.java:58) at com.inversoft.scheduler.InjectableRunnable.run(InjectableRunnable.java:56) at com.inversoft.scheduler.DefaultScheduler.<init>(DefaultScheduler.java:70) at com.inversoft.scheduler.DefaultScheduler$$FastClassByGuice$$208640461.GUICE$TRAMPOLINE(<generated>) at com.inversoft.scheduler.DefaultScheduler$$FastClassByGuice$$208640461.apply(<generated>) at com.google.inject.internal.DefaultConstructionProxyFactory$FastClassProxy.newInstance(DefaultConstructionProxyFactory.java:82) at com.google.inject.internal.ConstructorInjector.provision(ConstructorInjector.java:114) at com.google.inject.internal.ConstructorInjector.construct(ConstructorInjector.java:91) at com.google.inject.internal.ConstructorBindingImpl$Factory.get(ConstructorBindingImpl.java:300) at com.google.inject.internal.ProviderToInternalFactoryAdapter.get(ProviderToInternalFactoryAdapter.java:40) at com.google.inject.internal.SingletonScope$1.get(SingletonScope.java:169) at com.google.inject.internal.InternalFactoryToProviderAdapter.get(InternalFactoryToProviderAdapter.java:45) at com.google.inject.internal.FactoryProxy.get(FactoryProxy.java:60) at com.google.inject.internal.InternalInjectorCreator.loadEagerSingletons(InternalInjectorCreator.java:213) at com.google.inject.internal.InternalInjectorCreator.injectDynamically(InternalInjectorCreator.java:186) at com.google.inject.internal.InternalInjectorCreator.build(InternalInjectorCreator.java:113) at com.google.inject.Guice.createInjector(Guice.java:87) at com.google.inject.Guice.createInjector(Guice.java:69) at com.google.inject.Guice.createInjector(Guice.java:59) at org.primeframework.mvc.guice.GuiceBootstrap.initialize(GuiceBootstrap.java:60) at org.primeframework.mvc.BasePrimeMain.hup(BasePrimeMain.java:69) at org.primeframework.mvc.BasePrimeMain.start(BasePrimeMain.java:100) at io.fusionauth.app.FusionAuthMain.main(FusionAuthMain.java:27) Caused by: java.security.InvalidKeyException: Could not decode the X.509 public key. Expected at 2 values in the DER encoded sequence but found [2] at io.fusionauth.pem.PEMDecoder.decode_X_509(PEMDecoder.java:334) at io.fusionauth.pem.PEMDecoder.decode(PEMDecoder.java:112) ... 26 common frames omitted
2025-03-12T09:14:15.700435663Z
2025-03-12 09:14:15.699 AM INFO io.fusionauth.api.service.system.NodeService - Node [deb1186a-32c4-493e-83a7-a5299b66f6a8] promoted to master at [2025-03-12T09:14:15.699888323Z], the previous master Node [cf39ac34-f49e-4394-8721-8a2f8bfad22c] has been shutdown or removed
2025-03-12T09:14:16.467362998Z
2025-03-12 09:14:16.466 AM INFO io.fusionauth.app.primeframework.FusionHTTPContextAuthSetup - Initializing the FusionAuth HTTP Context.
2025-03-12T09:14:16.674891576Z
2025-03-12 09:14:16.674 AM INFO org.primeframework.mvc.PrimeMVCRequestHandler - Initializing Prime
2025-03-12T09:14:16.676199576Z
2025-03-12 09:14:16.675 AM INFO org.primeframework.mvc.PrimeMVCRequestHandler - Initializing Prime
2025-03-12T09:14:16.676563276Z
2025-03-12 09:14:16.676 AM INFO io.fusionauth.http.server.HTTPServer - Starting the HTTP server. Buckle up!
2025-03-12T09:14:16.684964236Z
2025-03-12 09:14:16.684 AM INFO io.fusionauth.http.server.HTTPServer - HTTP server listening on port [80]
2025-03-12T09:14:16.684994656Z
2025-03-12 09:14:16.684 AM INFO io.fusionauth.http.server.HTTPServer - HTTP server started successfully
2025-03-12T09:14:16.685016076Z
2025-03-12 09:14:16.684 AM INFO io.fusionauth.http.server.HTTPServer - Starting the HTTP server. Buckle up!
2025-03-12T09:14:16.685700126Z
2025-03-12 09:14:16.685 AM INFO io.fusionauth.http.server.HTTPServer - HTTP server listening on port [9012]
2025-03-12T09:14:16.685722766Z
2025-03-12 09:14:16.685 AM INFO io.fusionauth.http.server.HTTPServer - HTTP server started successfully