How to tune configuration

davidmw

Is there any documentation suggesting the best way to tune configuration settings?

We are getting some very long authentication response times (approaching 60 seconds) when we are hit with a several dozen login attempts within a small window.

Our status report has these interesting bits. I'm wondering if we should start by increasing the HikariPool Connections:

       "prime-mvc.[/oauth2/authorize].requests": {
            "count": 19859,
            "fifteenMinuteRate": 0.005875671297513242,
            "fiveMinuteRate": 0.0070476817956699606,
            "maximum": 118.563591,
            "mean": 11.500318605939524,
            "meanRate": 0.012490419029108284,
            "median": 5.936341,
            "minimum": 1.114009,
            "ninetyEighthPercentile": 17.961522,
            "ninetyFifthPercentile": 17.961522,
            "ninetyNinePointNinthPercentile": 17.961522,
            "ninetyNinthPercentile": 17.961522,
            "oneMinuteRate": 0.011810555622388075,
            "seventyFifthPercentile": 17.961522,
            "standardDeviation": 5.994036255601853,
            "unit": "MILLIS"
        },
        "prime-mvc.[/oauth2/callback].requests": {
            "count": 1417,
            "fifteenMinuteRate": 3.227927462601526E-4,
            "fiveMinuteRate": 8.17287897914934E-5,
            "maximum": 651.606361,
            "mean": 503.295733,
            "meanRate": 8.966385934082114E-4,
            "median": 503.295733,
            "minimum": 201.273076,
            "ninetyEighthPercentile": 503.295733,
            "ninetyFifthPercentile": 503.295733,
            "ninetyNinePointNinthPercentile": 503.295733,
            "ninetyNinthPercentile": 503.295733,
            "oneMinuteRate": 1.4771713901100558E-10,
            "seventyFifthPercentile": 503.295733,
            "standardDeviation": 5.3339365024206594E-36,
            "unit": "MILLIS"
        },
        "prime-mvc.[/oauth2/logout].requests": {
            "count": 373,
            "fifteenMinuteRate": 4.909154955918395E-11,
            "fiveMinuteRate": 1.6296978413163162E-26,
            "maximum": 13.484992,
            "mean": 3.474753371153825,
            "meanRate": 2.34803423303815E-4,
            "median": 3.476825,
            "minimum": 1.39451,
            "ninetyEighthPercentile": 3.476825,
            "ninetyFifthPercentile": 3.476825,
            "ninetyNinePointNinthPercentile": 3.476825,
            "ninetyNinthPercentile": 3.476825,
            "oneMinuteRate": 5.3386313991085494E-120,
            "seventyFifthPercentile": 3.476825,
            "standardDeviation": 0.06355696818827848,
            "unit": "MILLIS"
        },
        "prime-mvc.[/oauth2/passwordless].requests": {
            "count": 1005,
            "fifteenMinuteRate": 7.764998593245433E-6,
            "fiveMinuteRate": 1.1363578388340437E-9,
            "maximum": 98.271936,
            "mean": 27.053868,
            "meanRate": 6.368075801152446E-4,
            "median": 27.053868,
            "minimum": 0.159001,
            "ninetyEighthPercentile": 27.053868,
            "ninetyFifthPercentile": 27.053868,
            "ninetyNinePointNinthPercentile": 27.053868,
            "ninetyNinthPercentile": 27.053868,
            "oneMinuteRate": 7.67593180825067E-35,
            "seventyFifthPercentile": 27.053868,
            "standardDeviation": 1.3510365962356798E-38,
            "unit": "MILLIS"
        },
        "prime-mvc.[/oauth2/redirect].requests": {
            "count": 1054,
            "fifteenMinuteRate": 3.1745750077687547E-4,
            "fiveMinuteRate": 7.77428296785021E-5,
            "maximum": 2.695219,
            "mean": 2.695219,
            "meanRate": 6.669408482311662E-4,
            "median": 2.695219,
            "minimum": 0.680205,
            "ninetyEighthPercentile": 2.695219,
            "ninetyFifthPercentile": 2.695219,
            "ninetyNinePointNinthPercentile": 2.695219,
            "ninetyNinthPercentile": 2.695219,
            "oneMinuteRate": 1.1504222353483872E-10,
            "seventyFifthPercentile": 2.695219,
            "standardDeviation": 1.4570802492890024E-37,
            "unit": "MILLIS"
        },
        "prime-mvc.[/oauth2/token].requests": {
            "count": 1763,
            "fifteenMinuteRate": 3.2639949382991793E-4,
            "fiveMinuteRate": 8.44989963988031E-5,
            "maximum": 32.340534,
            "mean": 14.275601,
            "meanRate": 0.0011088520740279038,
            "median": 14.275601,
            "minimum": 8.287361,
            "ninetyEighthPercentile": 14.275601,
            "ninetyFifthPercentile": 14.275601,
            "ninetyNinePointNinthPercentile": 14.275601,
            "ninetyNinthPercentile": 14.275601,
            "oneMinuteRate": 1.7450718032937355E-10,
            "seventyFifthPercentile": 14.275601,
            "standardDeviation": 8.947911432912947E-38,
            "unit": "MILLIS"
        },
        "prime-mvc.[/oauth2/userinfo].requests": {
            "count": 1481,
            "fifteenMinuteRate": 3.26399770665232E-4,
            "fiveMinuteRate": 8.44989963988031E-5,
            "maximum": 39.8489,
            "mean": 6.378245,
            "meanRate": 9.384833641716966E-4,
            "median": 6.378245,
            "minimum": 0.342103,
            "ninetyEighthPercentile": 6.378245,
            "ninetyFifthPercentile": 6.378245,
            "ninetyNinePointNinthPercentile": 6.378245,
            "ninetyNinthPercentile": 6.378245,
            "oneMinuteRate": 1.7450718032937355E-10,
            "seventyFifthPercentile": 6.378245,
            "standardDeviation": 5.701224265924856E-38,
            "unit": "MILLIS"
        },

and

"gauges": {
           "HikariPool-1.pool.ActiveConnections": {
               "value": 0
           },
           "HikariPool-1.pool.IdleConnections": {
               "value": 10
           },
           "HikariPool-1.pool.MaxConnections": {
               "value": 10
           },
           "HikariPool-1.pool.MinConnections": {
               "value": 10
           },
           "HikariPool-1.pool.PendingConnections": {
               "value": 0
           },
           "HikariPool-1.pool.TotalConnections": {
               "value": 10
           }
       },
       "histograms": {
           "HikariPool-1.pool.ConnectionCreation": {
               "count": 67088,
               "maximum": 15,
               "mean": 5.0349683921225585,
               "median": 6.0,
               "minimum": 2,
               "ninetyEighthPercentile": 7.0,
               "ninetyFifthPercentile": 7.0,
               "ninetyNinePointNinthPercentile": 7.0,
               "ninetyNinthPercentile": 7.0,
               "seventyFifthPercentile": 7.0,
               "standardDeviation": 1.786761425473517
           },
           "HikariPool-1.pool.Usage": {
               "count": 840922,
               "maximum": 48,
               "mean": 1.583941103789779,
               "median": 1.0,
               "minimum": 0,
               "ninetyEighthPercentile": 8.0,
               "ninetyFifthPercentile": 6.0,
               "ninetyNinePointNinthPercentile": 8.0,
               "ninetyNinthPercentile": 8.0,
               "seventyFifthPercentile": 3.0,
               "standardDeviation": 1.8982617023833674
           }
       },

dan

Hiya,

There's not a guide to tune configurations (though there is this troubleshooting section of the docs).

A few more questions to start off:

What version of FusionAuth are you running and how are you running it (k8s, etc)?
How many nodes are you running?
Have you customized it (amount of memory, any other settings)?
What database engine (mysql, postgresql, etc) are you running?
Are you seeing any error messages in the FusionAuth logs?
Do you have metrics on other parts of the system (database, load balancer)? Are they showing stress? Is the server FusionAuth is running on pegged in terms of i/o or CPU or memory?
Are these users all on the same application? Are the login attempts succeeding or failing?
Can you easily replicate the long auth durations (with a load testing script or curl scripts)?

At first blush it looks like you don't need more connections, just because 99th percentile only uses 10 connections, but it's hard to judge without knowing more context.

davidmw

Hi Dan,

Running single node of 1.18.5 on Kubernetes with postgresql on the same VM. We don't see the problem very often so let me get some load testing configured.

David