We have some python scripts that we use to get all the users in fusionauth. There are less than 600 users. If we try to get them all at one time or too large a page size fusionauth crashes with an out of memory error and is non responsive. We tried upping our memory on the server but still crashes. We are running it as a docker container in aws and tried with 1gb and 2gb of memory.
Once the error occurs we need to re run the task to get it back up. Is this expected behavior?

Hi,

We upgraded from 1.40.2 to 1.41.3 and are seeing massive performance issues. Everything is very slow. Signing in, clicking on any pages in the admin side takes several seconds. Doing an api call to get a user takes over 6 seconds.

The only error in the logs I am seeing is this one:

2022-11-25T07:54:57.944-07:00	java.io.IOException: Broken pipe
2022-11-25T07:54:57.944-07:00	at java.base/sun.nio.ch.FileDispatcherImpl.writev0(Native Method)
2022-11-25T07:54:57.944-07:00	at java.base/sun.nio.ch.SocketDispatcher.writev(SocketDispatcher.java:66)
2022-11-25T07:54:57.944-07:00	at java.base/sun.nio.ch.IOUtil.write(IOUtil.java:217)
2022-11-25T07:54:57.944-07:00	at java.base/sun.nio.ch.IOUtil.write(IOUtil.java:153)
2022-11-25T07:54:57.944-07:00	at java.base/sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:563)
2022-11-25T07:54:57.944-07:00	at java.base/java.nio.channels.SocketChannel.write(SocketChannel.java:642)
2022-11-25T07:54:57.944-07:00	at io.fusionauth.http.server.HTTPServerThread.write(HTTPServerThread.java:298)
2022-11-25T07:54:57.944-07:00	at io.fusionauth.http.server.HTTPServerThread.run(HTTPServerThread.java:158)

We run this in a docker container in aws. I have tried running more instances of it but that did not change anything. CPU load will get high if anyone tries to do more than one thing with the system. Memory stays the same. Any ideas on what we can do or what changed?

@dan Issue created here:
https://github.com/FusionAuth/fusionauth-issues/issues/1717

I would like to add that we have a production instance that is running with less resources with no issues. The production instance has way less users, groups, registrations, applications, etc. So the data seems to have something to do with it.

Hi,

We upgraded from 1.40.2 to 1.41.3 and are seeing massive performance issues. Everything is very slow. Signing in, clicking on any pages in the admin side takes several seconds. Doing an api call to get a user takes over 6 seconds.

The only error in the logs I am seeing is this one:

2022-11-25T07:54:57.944-07:00	java.io.IOException: Broken pipe
2022-11-25T07:54:57.944-07:00	at java.base/sun.nio.ch.FileDispatcherImpl.writev0(Native Method)
2022-11-25T07:54:57.944-07:00	at java.base/sun.nio.ch.SocketDispatcher.writev(SocketDispatcher.java:66)
2022-11-25T07:54:57.944-07:00	at java.base/sun.nio.ch.IOUtil.write(IOUtil.java:217)
2022-11-25T07:54:57.944-07:00	at java.base/sun.nio.ch.IOUtil.write(IOUtil.java:153)
2022-11-25T07:54:57.944-07:00	at java.base/sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:563)
2022-11-25T07:54:57.944-07:00	at java.base/java.nio.channels.SocketChannel.write(SocketChannel.java:642)
2022-11-25T07:54:57.944-07:00	at io.fusionauth.http.server.HTTPServerThread.write(HTTPServerThread.java:298)
2022-11-25T07:54:57.944-07:00	at io.fusionauth.http.server.HTTPServerThread.run(HTTPServerThread.java:158)

We run this in a docker container in aws. I have tried running more instances of it but that did not change anything. CPU load will get high if anyone tries to do more than one thing with the system. Memory stays the same. Any ideas on what we can do or what changed?

@dan Issue created here:
https://github.com/FusionAuth/fusionauth-issues/issues/1717

@dan I did not have FUSIONAUTH_APP_MEMORY set so it was just at the default value.
I upped the value to 1gb when the container had 1gb and it no longer crashed with out of memory errors however the container would still die then restart. Not seeing any logs for it.
I tried making the value smaller than 1gb but it behaved the same. I then upped the container to 2 gb of memory and set FUSIONAUTH_APP_MEMORY to 1gb. Now it seems to be working ok and not crashing. Is there some further performance tuning that I can do on it?

@dan
We are running 1.36.4 Development edition. Haven't tried it on our production instance as we have not moved to production yet.

This is basically the query we are running:

    query_string = {'queryString': '*', 'numberOfResults': 600}
    search_string = {'search': query_string}
    search_response = fusion_auth_client.search_users_by_query(search_string)

I see this error:

An exception occurred while managing an async task.

Exception:
java.lang.Exception: java.lang.OutOfMemoryError: Java heap space
	at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
	at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:77)
	at java.base/jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
	at java.base/java.lang.reflect.Constructor.newInstanceWithCaller(Constructor.java:499)
	at java.base/java.lang.reflect.Constructor.newInstance(Constructor.java:480)
	at org.mybatis.guice.transactional.TransactionalMethodInterceptor.convertThrowableIfNeeded(TransactionalMethodInterceptor.java:171)
	at org.mybatis.guice.transactional.TransactionalMethodInterceptor.invoke(TransactionalMethodInterceptor.java:103)
	at org.mybatis.guice.transactional.TransactionalMethodInterceptor.invoke(TransactionalMethodInterceptor.java:100)
	at org.mybatis.guice.transactional.TransactionalMethodInterceptor.invoke(TransactionalMethodInterceptor.java:100)
	at org.mybatis.guice.transactional.TransactionalMethodInterceptor.invoke(TransactionalMethodInterceptor.java:100)
	at io.fusionauth.api.service.system.DefaultAsyncTaskManager.run(DefaultAsyncTaskManager.java:169)
	at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539)
	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
	at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
	at java.base/java.lang.Thread.run(Thread.java:833)
Caused by: java.lang.OutOfMemoryError: Java heap space

The user.data has some information but I wouldn't say it is a ton Something like:

accountId
1234
testAccount
Y
Application1
applicationId
x9999xxx-999x-99xx-xx9x-9x999x9xxx99
groupId
999999x9-9x99-9xx9-9999-xx999xxx99xx
testData
test

I don't see anything else in the logs about it.

25 users seems ok. Another developer did the work on it. I am not sure if he tried numbers in between 25 and 600. 600 makes it fail for sure.

The database is a Aurora PostgreSQL Serverless and this is the only thing on it. Load is very low on it. The CPU is between 15% and 20% and never really changes much.

@dan database

When doing a patch or put to a group to update it's data attribute we are running into an issue. The application roles are getting lost. This happens whether or not we include the roles. The operation returns as a success and includes the role ids but doing a get on the group afterwards shows that the roles are all gone. This is verified by looking at the group in fusionauth.

We have some python scripts that we use to get all the users in fusionauth. There are less than 600 users. If we try to get them all at one time or too large a page size fusionauth crashes with an out of memory error and is non responsive. We tried upping our memory on the server but still crashes. We are running it as a docker container in aws and tried with 1gb and 2gb of memory.
Once the error occurs we need to re run the task to get it back up. Is this expected behavior?

@joshua Thanks for confirming the behavior I am seeing. I will try using a library to validate the jwt instead.

Hi,

I am trying to implement the client credentials grant for an API.
I have a paid version of fusionAuth.
I have created two entity types for it: An API and an API client.
I created two entities one for each of the types.
I have the api client entity a grant to the api entity with the permissions that are on the entity type.

Using postman I am able to do a client credential grant and get a token back. When I base 64 decode the token I see the expected permissions from the target entity.

My problem is when I am trying to introspect the token in postman. I am getting this response back:

{
    "active": false
}

I am passing the token and the client_id as parameters to the introspection url. Why is it returning false?

I have an application and if I do an authorization grant for a user I am able to introspect that token and it returns active as being true and the additional data.

Is there some setup or something that I am missing?