FusionAuth
    • Home
    • Categories
    • Recent
    • Popular
    • Pricing
    • Contact us
    • Docs
    • Login

    Performance issues even with a 8 Core + 32 gigs.

    Scheduled Pinned Locked Moved
    Q&A
    java performance
    4
    21
    71.6k
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • S
      sswami
      last edited by

      Oh! Yes, we have tried that much earlier, sorry didn't tell you... we are out of Memory Heap... Right now its taking 100% CPU.

      1 Reply Last reply Reply Quote 0
      • robotdanR
        robotdan
        last edited by robotdan

        If your system is cpu bound then you need to scale this horizontally. Each node will only be able to hash n passwords per second. Once you reach this limit, the only way to perform more hashes per second is to reduce the hash complexity or add more nodes. Reducing the has complexity is not recommend as it makes these hashes easier to brute force.

        It is clear that you have a lot of knowledge of your environment and have invested a lot of time into this solution. We have many large scale production instances, I'm confident FusionAuth can scale to your requirements. However as I'm sure you're aware performance tuning is not a simple equation.

        There are a lot of possible variables here. The best way for you to get this tuned and ready for production is to purchase support so you can get engineering support.

        S 1 Reply Last reply Reply Quote 0
        • S
          sswami @robotdan
          last edited by sswami

          @robotdan Thank you very much for your reply... Well, this is 1 time but although Please let me know where to purchase for the support and a direct link to the suited package shall be appreciated.

          Moreover,

          1. Why is just rendering the SSO page taking so long?, Password hashing is far story...
          2. We have completely reduced crypto to Factor=2 with SHA-256 but still, an 8core CPU is reaching 100% for about 25-30 TPUs
          3. We are trying the "creating nodes" way.
          4. Also locally trying to profile FusionAuth Process Stack.
          5. Also, please favour us by telling on a High level / Approximation if there is nothing running on the VM and it's only to load FusionAuth SSO, what should be the best Performance expected. I agree, there must be some configs, threads, workers into the equation. But If you were to optimize all those, what would you achieve on an approximation. This will help us understand if its the Limitation by the server resources (CPU/RAM/NODES) or its simply some misconfiguration somewhere.
          1 Reply Last reply Reply Quote 0
          • S
            sjswami
            last edited by

            We performed load testing using jMeter tool in a small machine to do CPU Sampling profiling. Configuration core i5 4 core 3.6Ghz, 16GB RAM. To Fusionauth server 8GB was allocated. DB Mysql 8.0.21. Fresh DB setup, no user except admin.

            Performance testing scenario
            25 users, each user loading 100 times login page http:localhost:9011/oauth2/authorize?client_id=30d6e7be-407d-..

            Results
            With Fusionauth Server 1.17.0 - 25 page/sec
            With old Fusionauth Server 1.6.0 - 112 page/sec

            On 1.17.0 Server we did cpu sampling and observer this.
            57300dbc-25b2-4932-a4f3-848c05a6ea96-image.png

            Seems that google library is causing the issue. While older server was performing better.

            Moreover I could no upgrade to new version 1.19.2 from 1.17.0
            Below is the exception log
            -- Update the version
            UPDATE version
            SET version = '1.19.0';
            . Cause: java.sql.SQLSyntaxErrorException: You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near 'DELIMITER $$
            CREATE FUNCTION generate_id()
            RETURNS BINARY(16)
            NOT DETERMI' at line 1
            at org.apache.ibatis.jdbc.ScriptRunner.executeFullScript(ScriptRunner.java:133)
            at org.apache.ibatis.jdbc.ScriptRunner.runScript(ScriptRunner.java:108)
            at com.inversoft.maintenance.db.SQLExecutor.executeSQLScriptWithError(SQLExecutor.java:43)
            at com.inversoft.maintenance.db.JDBCMaintenanceModeDatabaseService.upgrade(JDBCMaintenanceModeDatabaseService.java:337)
            at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
            at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
            at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:630)
            ... 1 common frames omitted
            Caused by: java.sql.SQLSyntaxErrorException: You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near 'DELIMITER $$
            CREATE FUNCTION generate_id()
            RETURNS BINARY(16)
            NOT DETERMI' at line 1
            at com.mysql.cj.jdbc.exceptions.SQLError.createSQLException(SQLError.java:120)
            at com.mysql.cj.jdbc.exceptions.SQLError.createSQLException(SQLError.java:97)
            at com.mysql.cj.jdbc.exceptions.SQLExceptionsMapping.translateException(SQLExceptionsMapping.java:122)
            at com.mysql.cj.jdbc.StatementImpl.executeInternal(StatementImpl.java:764)
            at com.mysql.cj.jdbc.StatementImpl.execute(StatementImpl.java:648)
            at org.apache.ibatis.jdbc.ScriptRunner.executeStatement(ScriptRunner.java:236)
            at org.apache.ibatis.jdbc.ScriptRunner.executeFullScript(ScriptRunner.java:128)
            ... 7 common frames omitted

            Looking at the exception, the migration script of 1.19.0 has syntex error "DELIMITER $$" in Mysql 8.0.21.

            1 Reply Last reply Reply Quote 0
            • danD
              dan
              last edited by

              Hiya,

              You can purchase a support plan here: https://fusionauth.io/pricing (you'll want either 'enterprise' or 'premium' as the 'developer' plan doesn't include support).

              I think the bug you are seeing around the SQL delimiter was fixed in 1.19.5, so if you want to upgrade to 1.19.x, I'd recommend upgrading to at least 1.19.5. (The bug is here: https://github.com/FusionAuth/fusionauth-issues/issues/859 but in the comments on https://github.com/FusionAuth/fusionauth-issues/issues/862 Daniel mentions removing that code.)

              --
              FusionAuth - Auth for devs, built by devs.
              https://fusionauth.io

              1 Reply Last reply Reply Quote 0
              • S
                sjswami
                last edited by

                Thank you,

                After upgrading to 1.19.6, there was no error in db migraion with mysql 8.x. But I was unable to load the fusionauth dashboard. After successfull login it shows like c400faa2-319a-4df6-8053-8ccbe0d38563-image.png

                Then I alter the applications table -> FusionAuth. I changed the data column
                From
                {"data": {}, "jwtConfiguration": {"enabled": false, "timeToLiveInSeconds": 0, "refreshTokenTimeToLiveInMinutes": 0}, "loginConfiguration": {"allowTokenRefresh": false, "generateRefreshTokens": false, "requireAuthentication": true}, "oauthConfiguration": {"clientId": "3c219e58-ed0e-4b18-ad48-f4f92793ae32", "logoutURL": "/", "clientSecret": "Zj...", "enabledGrants": ["authorization_code"], "generateRefreshTokens": true, "authorizedRedirectURLs": ["/login"], "requireClientAuthentication": true}, "verifyRegistration": false, "samlv2Configuration": {"debug": false, "enabled": false, "xmlSignatureC14nMethod": "exclusive_with_comments"}, "passwordlessConfiguration": {"enabled": false}, "registrationConfiguration": {"type": "basic", "enabled": false, "fullName": {"enabled": false, "required": false}, "lastName": {"enabled": false, "required": false}, "birthDate": {"enabled": false, "required": false}, "firstName": {"enabled": false, "required": false}, "middleName": {"enabled": false, "required": false}, "loginIdType": "email", "mobilePhone": {"enabled": false, "required": false}, "confirmPassword": false}, "authenticationTokenConfiguration": {"enabled": false}}

                To

                {"jwtConfiguration": {"enabled": true, "timeToLiveInSeconds": 60, "refreshTokenExpirationPolicy": "SlidingWindow", "refreshTokenTimeToLiveInMinutes": 60, "refreshTokenUsagePolicy": "Reusable"},"registrationConfiguration": {"type":"basic"}, "oauthConfiguration": {"authorizedRedirectURLs": ["/login"], "clientId": "3c219e58-ed0e-4b18-ad48-f4f92793ae32", "clientSecret": "Zm...=", "enabledGrants": ["authorization_code"], "logoutURL": "/", "generateRefreshTokens": true, "requireClientAuthentication": true},"loginConfiguration": {"allowTokenRefresh": false, "generateRefreshTokens": false, "requireAuthentication": true}}

                Then it started loading.

                For now this migration issue is fixed.

                But the performance hostspot still exists with the latest version just loading login page.
                6d0f089d-987d-4150-8a65-5ee9dff7104b-image.png

                DB was performing well, the average execution was around 1ms
                3307f66a-322c-4637-b238-af90aa1f5663-image.png

                1 Reply Last reply Reply Quote 0
                • danD
                  dan
                  last edited by

                  Hiya,

                  Based on this message:

                  Please let me know where to purchase for the support and a direct link to the suited package shall be appreciated.

                  I believe you were planning to purchase support? If so, please open a support ticket and reference this forum post.

                  Otherwise I'd be interested to hear how your horizontal scaling efforts are going? And also you might consider switching the F class of Azure VMs, as those appear from the docs to be better suited for CPU intensive operations: https://azure.microsoft.com/en-us/blog/f-series-vm-size/

                  --
                  FusionAuth - Auth for devs, built by devs.
                  https://fusionauth.io

                  1 Reply Last reply Reply Quote 0
                  • S
                    sjswami
                    last edited by

                    We have already implemented horizontal scaling. Currently 4 nodes are running. So for now users are able to login.

                    I would be great if you can fix the google getInstance issue fix in upcoming releases.

                    Morover running migration 1.19.0 over large data set, Below migration was generating duplicate ids in refresh_tokens table
                    UPDATE refresh_tokens
                    SET id = SUBSTR(CONCAT(MD5(RAND()), MD5(RAND())), 3, 16)
                    WHERE id IS NULL;

                    No of rows where 60,000 and out of which 422 duplicate where generated. So Migration was failing. For now I have removed duplicate records, for those users new refresh token will be generated during login.

                    You may fix this issue, other customers may face this problem.

                    1 Reply Last reply Reply Quote 0
                    • robotdanR
                      robotdan
                      last edited by

                      Thanks for letting us know about the UK violations, I would have expected SUBSTR(CONCAT(MD5(RAND()), MD5(RAND())), 3, 16) to generated a unique value. We'll have to do some testing.

                      1 Reply Last reply Reply Quote 0
                      • danD
                        dan
                        last edited by

                        Thanks @sjswami , this duplicate ids issue is now resolved in 1.19.7. Appreciate you letting us know about it: https://github.com/FusionAuth/fusionauth-issues/issues/890

                        --
                        FusionAuth - Auth for devs, built by devs.
                        https://fusionauth.io

                        1 Reply Last reply Reply Quote 1
                        • First post
                          Last post