FusionAuth
    • Home
    • Categories
    • Recent
    • Popular
    • Pricing
    • Contact us
    • Docs
    • Login

    java.sql.SQLTransientConnectionException: HikariPool-1 - Connection is not available

    Scheduled Pinned Locked Moved Unsolved
    Q&A
    2
    7
    6.4k
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • M
      mark.shapiro
      last edited by

      We've been seeing this error pop up and cause some failures to our API calls recently. We're on an Essentials Hosted plan currently on v1.46.0.

      2023-10-20 08:07:06.942 PM ERROR io.fusionauth.api.service.system.DefaultAsyncTaskManager - An exception occurred while managing an async task.
      org.apache.ibatis.exceptions.PersistenceException: 
      ### Error querying database.  Cause: java.sql.SQLTransientConnectionException: HikariPool-1 - Connection is not available, request timed out after 2000ms.
      ### The error may exist in io/fusionauth/api/domain/LockMapper.java (best guess)
      ### The error may involve io.fusionauth.api.domain.LockMapper.lock
      ### The error occurred while executing a query
      ### Cause: java.sql.SQLTransientConnectionException: HikariPool-1 - Connection is not available, request timed out after 2000ms.
      	at org.apache.ibatis.exceptions.ExceptionFactory.wrapException(ExceptionFactory.java:30)
      

      I saw this related issue and mentioned about IOPS limit, but I haven't seen that metric anywhere. Is this something about how we're using FusionAuth or just a system issue? We've been ramping up more users onto the system recently, along with API calls so not sure if it's something we're doing, but wanted to make sure this doesn't become a larger issue as we continue to scale.

      mark.robustelliM 1 Reply Last reply Reply Quote 0
      • mark.robustelliM
        mark.robustelli @mark.shapiro
        last edited by

        @mark-shapiro , just wanted you to be aware that since you have the Essentials plan, you can get support directly thorough the FusionAuth Account Portal if this is time sensitive for you.

        M 1 Reply Last reply Reply Quote 0
        • M
          mark.shapiro @mark.robustelli
          last edited by

          @mark-robustelli It's not today critical, though I logged into the console and I don't have a support option (I might have to bug our admin to go that route).

          Though I also see Essentials is under Self-Hosted, so now I'm questioning life. I'll have to check with our DevOps team as well. So will do that (Monday) if there's there anything to look at on our side, let me know.

          mark.robustelliM 1 Reply Last reply Reply Quote 0
          • mark.robustelliM
            mark.robustelli @mark.shapiro
            last edited by

            @mark-shapiro Please do check with your team. There are two different things, the licensing and the hosting. It is possible to have Business Hosting or Self-Hosting with the Essentials Plan.

            M 1 Reply Last reply Reply Quote 0
            • M
              mark.shapiro @mark.robustelli
              last edited by

              @mark-robustelli I verified with the domain name itself.

              nslookup login.mycompany.com
              
              Server:         8.8.8.8
              Address:        8.8.8.8#53
              
              Non-authoritative answer:
              login.mycompany.com       canonical name = mycompany-prod.fusionauth.io.
              

              So should confirm we're using hosted (and had been what I thought).

              mark.robustelliM 1 Reply Last reply Reply Quote 0
              • mark.robustelliM
                mark.robustelli @mark.shapiro
                last edited by

                @mark-shapiro , Thanks for clearing up the hosting. It sounds like you have the business hosting plan. Now, which FusionAuth license (This is different than the hosting) do you have? In an earlier post it sounded like you had the Essentials plan. Can you please confirm that with your admin? (It should be one of the following: Community, Starter, Essentials, or Enterprise)

                M 1 Reply Last reply Reply Quote 0
                • M
                  mark.shapiro @mark.robustelli
                  last edited by

                  @mark-robustelli I had opened a support ticket (via email) and got this response

                  If you have external calls in your integration, you will want to ensure a fast response  
                  
                  Lambdas - If you are calling FusionAuth APIs in a lambda, ensure a connection over port 9012 (as opposed to port 9011).  Any other external HTTP calls (to your own endpoints) using HTTPConnect should return quickly as well to ensure optimal performance. 
                  
                  Connectors - If you have a connector, then FusionAuth will hold things in flight (database connections, in-memory information, etc) while we wait for your connector to return an authentication response and log the user in.    
                  
                  Ideally, your connector would have a read timeout of 1ms and a connect timeout of 2ms or less.   
                  
                  Higher values mean FusionAuth will have fewer resources (database connections, etc) available to service incoming login requests (as older login requests are still in flight waiting for the Connector return). 
                  
                  Webhooks -  
                  
                  If you have any webhooks, these should also return quickly to optimize performance.  Any SocketTimeoutException (read or connect) in the Event Log would indicate a slower webhook integration. 
                  
                  Adjust Period Tasks  
                  
                  If you have any periodic tasks running against your deployment (perhaps to synchronize user data, application data, etc by calling our APIs), then you will want to write back-off logic if the system shows a heavy load (monitoring documentation). 
                  
                  Turn off logging in production  
                  
                  This change will be less impactful but will help nevertheless.  Logging should be used when there is an issue with SMTP/email, lambdas, connectors, users, etc but turned off in production when not needed.  Logs are output to System > Event Log.  The debug enabled toggle throughout the UI (and API) indicates whether this logging will occur.  
                  

                  The big ones were that we were making calls to the API in a lambda and switched the port as mentioned. We also had left on debug logging in a lot of places so have disabled that for now. Validated our webhooks are all running pretty fast (median type is 1-2ms) and connector is pretty fast, though slower 20-30ms median.

                  1 Reply Last reply Reply Quote 0
                  • First post
                    Last post