Admin UI actions over themes blocks application since 1.61.2

mark.robustelli

@david-cuen Thanks for the detailed message. I hope to have some time today to try and replicate this. I will let you know what I see.

mark.robustelli

@david-cuen can you try to spin up a new instance and let me know if you see the problem? If not, it may have to do specifically with the themes themselves. If the new instance works correctly, import each theme and test. This may help narrow the issue.

david.cuen

Hi @mark-robustelli, thanks for your reply.
I have created a new instance as you suggested.

With only the two default themes ("FusionAuth" and "FusionAuth - Simple"), everything works correctly.

I have been importing our themes one by one (all of them are advanced themes).
The search works correctly until it reaches 10 registered themes. After that number, the same error occurs.
It does not matter which themes are registered, but rather the total number of themes. Changing the themes creation order produces the same result.

To discard problems in our themes data, I deleted them and started testing from scratch.

I created new themes using the default advanced theme "FusionAuth" as base and duplicating it.
When the 10 existing themes are reached, the problem occurs again.

Creating new themes using the default simple theme as base, I have not been able to reproduce the problem exceeding 10 registered themes.

I assume that the problem is not related to the number of themes registered, but rather to the volume of data that they all contain.

mark.robustelli

@david-cuen Awesome that you have isolated the issues. I am going to test from my side and see if I can replicate with the 10+ themes. If I can then you should report this as a bug. I will let you know what I find. Hope to be able to get to this today.

mark.robustelli

@david-cuen OK, I was able to get more than 10, but you're right about it possibly being the size. Is it possible for you to share the theme you are using with us or me individually? I don't want you to give away any secrets, proprietary info or share anything private if you can't but getting a look at the theme may help.

david.cuen

@mark-robustelli I have downloaded the FusionAuth advanced default theme (id 75a068fd-e94b-451a-9aeb-3ddb9a3b5987) used in the tests with the FusionAuth CLI

The files are available in
https://github.com/davidcuenavoristravel/fusionauth-shared-themes/tree/main/default_theme/tpl

Best regards.

mark.robustelli

@david-cuen perfect. I was copying themes, but will us that one and let you know what I find.

mark.robustelli

@david-cuen Goodnews and badnews. The goodnews is that I imported the exact theme you pointed to and was able to copy that 11 times and not issues. My environment now has 26 themes. The badnews is that is not helping you. Sorry. I think we need to dig a little further. Based on the additional pages you mentioned (adding/editing tenant/application) I am wondering if something specific with the edits you have made to the themes. The fact that you change the order they are loaded in makes that less likely though. What are the sizes of the tpl folders for your themes?

david.cuen

@mark-robustelli The size of the tpl folders for each theme is around 352 KB - 404 KB.

I would like to mention that the hang occurs when launching the instance in a Linux environment with the Docker image.
If I launch a local instance on a Windows computer against the same PostgreSQL database, the hang problem does not occur.

Version: 1.62.1
Platform: Windows 11 10.0 amd64

Instances are launched with ENV var:
FUSIONAUTH_APP_MEMORY=1024M
DATABASE_CONNECTION_HEALTHCHECK_99PERCENT_MS=500

When consulting Prometheus metrics with /api/status after the instance has recovered from the hang, errors appear in the theme search

            "prime-mvc.[/api/theme/search].errors": {
                "count": 2,
                "fifteenMinuteRate": 0.001224500060165843,
                "fiveMinuteRate": 0.002437052376257949,
                "meanRate": 0.0012614192847508726,
                "oneMinuteRate": 0.003282810949244636
            },

Any idea how to find the error that occurs in the internal call from fusionauth admin app to its API?

david.cuen

@mark-robustelli I've launched the instance in the infrastructure enabling an agentlib to perform remote debugging of the JVM.

Debugging the theme search process, I see following exception produced in the RESTClient response of the API call

com.inversoft.rest.JSONException: Failed to parse the HTTP response as JSON. Actual HTTP response body:
Note: Output has been truncated to the first 1024 of 2560141 bytes.

{"themes":[{"data":{},"defaultMessages":"#\n# Date and Time formats\n#\ndate-format=M/d/yyyy\ndate-time-format=M/d/yyyy hh:mm a z\ndate-time-seconds-format=M/d/yyyy hh:mm:ss a z\n\n#\n# Text used on the page (inside the HTML). You can create new key-value pairs here and use them in the templates.\n#\naccess-denied=Access denied\naccount=Account\naction=Action\nadd-two-factor=Add two-factor\nadd-webauthn-passkey=Add passkey\nback-to-login=Return to Login\ncancel=Cancel\ncaptcha-google-branding=This site is protected by reCAPTCHA and the Google <a href=\"https://policies.google.com/privacy\" class=\"text-indigo-500 hover:text-indigo-700 font-medium focus:outline-none focus:underline\">Privacy Policy</a> and <a href=\"https://policies.google.com/terms\" class=\"text-indigo-500 hover:text-indigo-700 font-medium focus:outline-none focus:underline\">Terms of Service</a> apply.\ncreated=Created\ncustomize=Customize\nauthorized-not-registered=Registration is required to access this application and your account has no

Cause:

"com.fasterxml.jackson.databind.JsonMappingException: Premature EOF (through reference chain: io.fusionauth.domain.api.ThemeSearchResponse["themes"]->java.util.ArrayList[9])"

StackTrace:

com.inversoft.rest.JSONResponseHandler.apply(JSONResponseHandler.java:73)
com.inversoft.rest.RESTClient.go(RESTClient.java:430)
io.fusionauth.client.FusionAuthClient.searchThemes(FusionAuthClient.java:5372)
io.fusionauth.app.action.admin.theme.IndexAction.lambda$search$0(IndexAction.java:52)
io.fusionauth.client.LambdaDelegate.execute(LambdaDelegate.java:58)
io.fusionauth.app.action.admin.theme.IndexAction.search(IndexAction.java:52)
io.fusionauth.app.action.admin.BaseSearchAction.execute(BaseSearchAction.java:77)
java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103)
java.base/java.lang.reflect.Method.invoke(Method.java:580)
org.primeframework.mvc.util.ReflectionUtils.invoke(ReflectionUtils.java:443)
org.primeframework.mvc.action.DefaultActionInvocationWorkflow.execute(DefaultActionInvocationWorkflow.java:77)
org.primeframework.mvc.action.DefaultActionInvocationWorkflow.perform(DefaultActionInvocationWorkflow.java:60)
org.primeframework.mvc.workflow.SubWorkflowChain.continueWorkflow(SubWorkflowChain.java:50)
org.primeframework.mvc.message.DefaultMessageWorkflow.perform(DefaultMessageWorkflow.java:50)
org.primeframework.mvc.workflow.SubWorkflowChain.continueWorkflow(SubWorkflowChain.java:50)
org.primeframework.mvc.validation.DefaultValidationWorkflow.perform(DefaultValidationWorkflow.java:45)
org.primeframework.mvc.workflow.SubWorkflowChain.continueWorkflow(SubWorkflowChain.java:50)
org.primeframework.mvc.security.DefaultSecurityWorkflow.perform(DefaultSecurityWorkflow.java:79)
org.primeframework.mvc.workflow.SubWorkflowChain.continueWorkflow(SubWorkflowChain.java:50)
org.primeframework.mvc.parameter.DefaultPostParameterWorkflow.perform(DefaultPostParameterWorkflow.java:49)
org.primeframework.mvc.workflow.SubWorkflowChain.continueWorkflow(SubWorkflowChain.java:50)
org.primeframework.mvc.content.DefaultContentWorkflow.perform(DefaultContentWorkflow.java:74)
org.primeframework.mvc.workflow.SubWorkflowChain.continueWorkflow(SubWorkflowChain.java:50)
org.primeframework.mvc.parameter.DefaultParameterWorkflow.perform(DefaultParameterWorkflow.java:58)
org.primeframework.mvc.workflow.SubWorkflowChain.continueWorkflow(SubWorkflowChain.java:50)
org.primeframework.mvc.parameter.DefaultURIParameterWorkflow.perform(DefaultURIParameterWorkflow.java:92)
org.primeframework.mvc.workflow.SubWorkflowChain.continueWorkflow(SubWorkflowChain.java:50)
org.primeframework.mvc.scope.DefaultScopeRetrievalWorkflow.perform(DefaultScopeRetrievalWorkflow.java:50)
org.primeframework.mvc.workflow.SubWorkflowChain.continueWorkflow(SubWorkflowChain.java:50)
org.primeframework.mvc.action.DefaultActionMappingWorkflow.perform(DefaultActionMappingWorkflow.java:130)
org.primeframework.mvc.workflow.SubWorkflowChain.continueWorkflow(SubWorkflowChain.java:50)
org.primeframework.mvc.security.DefaultSavedRequestWorkflow.perform(DefaultSavedRequestWorkflow.java:65)
org.primeframework.mvc.workflow.SubWorkflowChain.continueWorkflow(SubWorkflowChain.java:50)
org.primeframework.mvc.cors.CORSFilter.doFilter(CORSFilter.java:188)
org.primeframework.mvc.cors.CORSRequestWorkflow.perform(CORSRequestWorkflow.java:66)
org.primeframework.mvc.workflow.SubWorkflowChain.continueWorkflow(SubWorkflowChain.java:50)
org.primeframework.mvc.workflow.DefaultMVCWorkflow.perform(DefaultMVCWorkflow.java:109)
org.primeframework.mvc.PrimeMVCRequestHandler.handle(PrimeMVCRequestHandler.java:76)
io.fusionauth.http.server.internal.HTTPWorker.run(HTTPWorker.java:183)
java.base/java.lang.VirtualThread.run(VirtualThread.java:329)

mark.robustelli

@david-cuen sorry for the delayed response. I was unexpectedly out. Thanks for trying to dig a bit deeper. The message you definitely leads me to believe that it is either a specific theme or as you said a size thing. Both are a bit confusing to me as the size you mentioned for your themes does not seem that bit and you stated that you tried loading the themes in different orders and that did not seem to make a difference. I will see if I can dig a little deeper on these error messages and see if there is anything I can find.