Performance issues even with a 8 Core + 32 gigs.

dan

Seems like you are doing all the right things.

@sswami said in Performance issues even with a 8 Core + 32 gigs.:

java.lang.OutOfMemoryError: Java heap space
at java.base/java.lang.reflect.Method.copy(Method.java:158)

If you have 32GB of memory, can you increase the amount available to FusionAuth beyond 2GB? Try 8 or 16. That should help with the heap space issue.

Can you try that and let me know if there are new errors in the FusionAuth log file?

sswami

@dan Surely... I shall try and revert.

However, Can I know one thing? What performance should we be expecting with this VM, like can you just give me approx. range? Or what best have you achieved as test load, how does that translate to our machine?

dan

Sorry, I don't understand: What do you mean by "I shall try and revert."?

I'm not aware of any load tests on this type of VM, but I'll ask the team and get back to you.

dan

Word from the team is we haven't load tested on a VM recently.

sswami

Hello @dan !

Any update on the Load test? Can you suggest what should be our server config & resources to handle 7000+ login requests at a time? We are still facing this issue? The fusionauth-app (https://login.gurukul.org/oauth2/authorize) Java Process consumes 100% CPU when more than around 1000 users tries to login in at a time.

Also, as we never had and are not having much regular concerns, we have not opted for support package. This issue if solved will the max we would need support actually.

Thank you...

sswami

Java Process consumes 100% CPU when more than around 1000 users tries to login in at a time
Its

ONLY a max of 20 clients per second and 1000 users over a min with an average response time of 10secs!

We have checked all possible things what we could hunt over the internet. But with our limited knowledge, we are unable to solve this.

dan

Did you try changing the memory allocated to FusionAuth, as I mentioned here: https://fusionauth.io/community/forum/topic/370/performance-issues-even-with-a-8-core-32-gigs/5

If you don't have enough memory and there are a large number of OutOfMemoryErrors, that could be causing the symptoms you are seeing.

sswami

Oh! Yes, we have tried that much earlier, sorry didn't tell you... we are out of Memory Heap... Right now its taking 100% CPU.

robotdan

If your system is cpu bound then you need to scale this horizontally. Each node will only be able to hash n passwords per second. Once you reach this limit, the only way to perform more hashes per second is to reduce the hash complexity or add more nodes. Reducing the has complexity is not recommend as it makes these hashes easier to brute force.

It is clear that you have a lot of knowledge of your environment and have invested a lot of time into this solution. We have many large scale production instances, I'm confident FusionAuth can scale to your requirements. However as I'm sure you're aware performance tuning is not a simple equation.

There are a lot of possible variables here. The best way for you to get this tuned and ready for production is to purchase support so you can get engineering support.

sswami

@robotdan Thank you very much for your reply... Well, this is 1 time but although Please let me know where to purchase for the support and a direct link to the suited package shall be appreciated.

Moreover,

Why is just rendering the SSO page taking so long?, Password hashing is far story...
We have completely reduced crypto to Factor=2 with SHA-256 but still, an 8core CPU is reaching 100% for about 25-30 TPUs
We are trying the "creating nodes" way.
Also locally trying to profile FusionAuth Process Stack.
Also, please favour us by telling on a High level / Approximation if there is nothing running on the VM and it's only to load FusionAuth SSO, what should be the best Performance expected. I agree, there must be some configs, threads, workers into the equation. But If you were to optimize all those, what would you achieve on an approximation. This will help us understand if its the Limitation by the server resources (CPU/RAM/NODES) or its simply some misconfiguration somewhere.

sjswami

We performed load testing using jMeter tool in a small machine to do CPU Sampling profiling. Configuration core i5 4 core 3.6Ghz, 16GB RAM. To Fusionauth server 8GB was allocated. DB Mysql 8.0.21. Fresh DB setup, no user except admin.

Performance testing scenario
25 users, each user loading 100 times login page http:localhost:9011/oauth2/authorize?client_id=30d6e7be-407d-..

Results
With Fusionauth Server 1.17.0 - 25 page/sec
With old Fusionauth Server 1.6.0 - 112 page/sec

On 1.17.0 Server we did cpu sampling and observer this.

Seems that google library is causing the issue. While older server was performing better.

Moreover I could no upgrade to new version 1.19.2 from 1.17.0
Below is the exception log
-- Update the version
UPDATE version
SET version = '1.19.0';
. Cause: java.sql.SQLSyntaxErrorException: You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near 'DELIMITER $$
CREATE FUNCTION generate_id()
RETURNS BINARY(16)
NOT DETERMI' at line 1
at org.apache.ibatis.jdbc.ScriptRunner.executeFullScript(ScriptRunner.java:133)
at org.apache.ibatis.jdbc.ScriptRunner.runScript(ScriptRunner.java:108)
at com.inversoft.maintenance.db.SQLExecutor.executeSQLScriptWithError(SQLExecutor.java:43)
at com.inversoft.maintenance.db.JDBCMaintenanceModeDatabaseService.upgrade(JDBCMaintenanceModeDatabaseService.java:337)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:630)
... 1 common frames omitted
Caused by: java.sql.SQLSyntaxErrorException: You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near 'DELIMITER $$
CREATE FUNCTION generate_id()
RETURNS BINARY(16)
NOT DETERMI' at line 1
at com.mysql.cj.jdbc.exceptions.SQLError.createSQLException(SQLError.java:120)
at com.mysql.cj.jdbc.exceptions.SQLError.createSQLException(SQLError.java:97)
at com.mysql.cj.jdbc.exceptions.SQLExceptionsMapping.translateException(SQLExceptionsMapping.java:122)
at com.mysql.cj.jdbc.StatementImpl.executeInternal(StatementImpl.java:764)
at com.mysql.cj.jdbc.StatementImpl.execute(StatementImpl.java:648)
at org.apache.ibatis.jdbc.ScriptRunner.executeStatement(ScriptRunner.java:236)
at org.apache.ibatis.jdbc.ScriptRunner.executeFullScript(ScriptRunner.java:128)
... 7 common frames omitted

Looking at the exception, the migration script of 1.19.0 has syntex error "DELIMITER $$" in Mysql 8.0.21.

dan

Hiya,

You can purchase a support plan here: https://fusionauth.io/pricing (you'll want either 'enterprise' or 'premium' as the 'developer' plan doesn't include support).

I think the bug you are seeing around the SQL delimiter was fixed in 1.19.5, so if you want to upgrade to 1.19.x, I'd recommend upgrading to at least 1.19.5. (The bug is here: https://github.com/FusionAuth/fusionauth-issues/issues/859 but in the comments on https://github.com/FusionAuth/fusionauth-issues/issues/862 Daniel mentions removing that code.)

sjswami

Thank you,

After upgrading to 1.19.6, there was no error in db migraion with mysql 8.x. But I was unable to load the fusionauth dashboard. After successfull login it shows like

Then I alter the applications table -> FusionAuth. I changed the data column
From
{"data": {}, "jwtConfiguration": {"enabled": false, "timeToLiveInSeconds": 0, "refreshTokenTimeToLiveInMinutes": 0}, "loginConfiguration": {"allowTokenRefresh": false, "generateRefreshTokens": false, "requireAuthentication": true}, "oauthConfiguration": {"clientId": "3c219e58-ed0e-4b18-ad48-f4f92793ae32", "logoutURL": "/", "clientSecret": "Zj...", "enabledGrants": ["authorization_code"], "generateRefreshTokens": true, "authorizedRedirectURLs": ["/login"], "requireClientAuthentication": true}, "verifyRegistration": false, "samlv2Configuration": {"debug": false, "enabled": false, "xmlSignatureC14nMethod": "exclusive_with_comments"}, "passwordlessConfiguration": {"enabled": false}, "registrationConfiguration": {"type": "basic", "enabled": false, "fullName": {"enabled": false, "required": false}, "lastName": {"enabled": false, "required": false}, "birthDate": {"enabled": false, "required": false}, "firstName": {"enabled": false, "required": false}, "middleName": {"enabled": false, "required": false}, "loginIdType": "email", "mobilePhone": {"enabled": false, "required": false}, "confirmPassword": false}, "authenticationTokenConfiguration": {"enabled": false}}

To

{"jwtConfiguration": {"enabled": true, "timeToLiveInSeconds": 60, "refreshTokenExpirationPolicy": "SlidingWindow", "refreshTokenTimeToLiveInMinutes": 60, "refreshTokenUsagePolicy": "Reusable"},"registrationConfiguration": {"type":"basic"}, "oauthConfiguration": {"authorizedRedirectURLs": ["/login"], "clientId": "3c219e58-ed0e-4b18-ad48-f4f92793ae32", "clientSecret": "Zm...=", "enabledGrants": ["authorization_code"], "logoutURL": "/", "generateRefreshTokens": true, "requireClientAuthentication": true},"loginConfiguration": {"allowTokenRefresh": false, "generateRefreshTokens": false, "requireAuthentication": true}}

Then it started loading.

For now this migration issue is fixed.

But the performance hostspot still exists with the latest version just loading login page.

DB was performing well, the average execution was around 1ms

dan

Hiya,

Based on this message:

Please let me know where to purchase for the support and a direct link to the suited package shall be appreciated.

I believe you were planning to purchase support? If so, please open a support ticket and reference this forum post.

Otherwise I'd be interested to hear how your horizontal scaling efforts are going? And also you might consider switching the F class of Azure VMs, as those appear from the docs to be better suited for CPU intensive operations: https://azure.microsoft.com/en-us/blog/f-series-vm-size/

sjswami

We have already implemented horizontal scaling. Currently 4 nodes are running. So for now users are able to login.

I would be great if you can fix the google getInstance issue fix in upcoming releases.

Morover running migration 1.19.0 over large data set, Below migration was generating duplicate ids in refresh_tokens table
UPDATE refresh_tokens
SET id = SUBSTR(CONCAT(MD5(RAND()), MD5(RAND())), 3, 16)
WHERE id IS NULL;

No of rows where 60,000 and out of which 422 duplicate where generated. So Migration was failing. For now I have removed duplicate records, for those users new refresh token will be generated during login.

You may fix this issue, other customers may face this problem.

robotdan

Thanks for letting us know about the UK violations, I would have expected SUBSTR(CONCAT(MD5(RAND()), MD5(RAND())), 3, 16) to generated a unique value. We'll have to do some testing.

dan

Thanks @sjswami , this duplicate ids issue is now resolved in 1.19.7. Appreciate you letting us know about it: https://github.com/FusionAuth/fusionauth-issues/issues/890