Fusionauth crashes after search for users

paul.fink

We have some python scripts that we use to get all the users in fusionauth. There are less than 600 users. If we try to get them all at one time or too large a page size fusionauth crashes with an out of memory error and is non responsive. We tried upping our memory on the server but still crashes. We are running it as a docker container in aws and tried with 1gb and 2gb of memory.
Once the error occurs we need to re run the task to get it back up. Is this expected behavior?

dan

@paul-fink Which search engine are you using, database or elasticsearch?

paul.fink

@dan database

dan

@paul-fink That should be plenty of memory.

What version of FusionAuth are you using?

What is the query you are running (a python snippet would be awesome)?

Are there any errors in the log files (either the event log or the docker STDOUT) that you can share?

Do you have a lot of data hanging off of the user object (in user.data)?

How large is too large a page of results? 25 users? 200 users?

How big is the database FusionAuth is using? Do you have any metrics for that? I don't think that is the issue, but if the database is thrashed, that could be causing some issues.

What edition are you running?

paul.fink

@dan
We are running 1.36.4 Development edition. Haven't tried it on our production instance as we have not moved to production yet.

This is basically the query we are running:

    query_string = {'queryString': '*', 'numberOfResults': 600}
    search_string = {'search': query_string}
    search_response = fusion_auth_client.search_users_by_query(search_string)

I see this error:

An exception occurred while managing an async task.

Exception:
java.lang.Exception: java.lang.OutOfMemoryError: Java heap space
	at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
	at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:77)
	at java.base/jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
	at java.base/java.lang.reflect.Constructor.newInstanceWithCaller(Constructor.java:499)
	at java.base/java.lang.reflect.Constructor.newInstance(Constructor.java:480)
	at org.mybatis.guice.transactional.TransactionalMethodInterceptor.convertThrowableIfNeeded(TransactionalMethodInterceptor.java:171)
	at org.mybatis.guice.transactional.TransactionalMethodInterceptor.invoke(TransactionalMethodInterceptor.java:103)
	at org.mybatis.guice.transactional.TransactionalMethodInterceptor.invoke(TransactionalMethodInterceptor.java:100)
	at org.mybatis.guice.transactional.TransactionalMethodInterceptor.invoke(TransactionalMethodInterceptor.java:100)
	at org.mybatis.guice.transactional.TransactionalMethodInterceptor.invoke(TransactionalMethodInterceptor.java:100)
	at io.fusionauth.api.service.system.DefaultAsyncTaskManager.run(DefaultAsyncTaskManager.java:169)
	at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539)
	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
	at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
	at java.base/java.lang.Thread.run(Thread.java:833)
Caused by: java.lang.OutOfMemoryError: Java heap space

The user.data has some information but I wouldn't say it is a ton Something like:

accountId
1234
testAccount
Y
Application1
applicationId
x9999xxx-999x-99xx-xx9x-9x999x9xxx99
groupId
999999x9-9x99-9xx9-9999-xx999xxx99xx
testData
test

I don't see anything else in the logs about it.

25 users seems ok. Another developer did the work on it. I am not sure if he tried numbers in between 25 and 600. 600 makes it fail for sure.

The database is a Aurora PostgreSQL Serverless and this is the only thing on it. Load is very low on it. The CPU is between 15% and 20% and never really changes much.

dan

@paul-fink Hmmm. Thank you for the stack traces. That is pretty clearly FusionAuth running out of memory.

You mentioned you are running with 1gb or 2gb of memory in the docker image.

What is the setting for FUSIONAUTH_APP_MEMORY?

paul.fink

@dan I did not have FUSIONAUTH_APP_MEMORY set so it was just at the default value.
I upped the value to 1gb when the container had 1gb and it no longer crashed with out of memory errors however the container would still die then restart. Not seeing any logs for it.
I tried making the value smaller than 1gb but it behaved the same. I then upped the container to 2 gb of memory and set FUSIONAUTH_APP_MEMORY to 1gb. Now it seems to be working ok and not crashing. Is there some further performance tuning that I can do on it?