Admin UI actions over themes blocks application since 1.61.2

david.cuen

@mark-robustelli I have downloaded the FusionAuth advanced default theme (id 75a068fd-e94b-451a-9aeb-3ddb9a3b5987) used in the tests with the FusionAuth CLI

The files are available in
https://github.com/davidcuenavoristravel/fusionauth-shared-themes/tree/main/default_theme/tpl

Best regards.

mark.robustelli

@david-cuen perfect. I was copying themes, but will us that one and let you know what I find.

mark.robustelli

@david-cuen Goodnews and badnews. The goodnews is that I imported the exact theme you pointed to and was able to copy that 11 times and not issues. My environment now has 26 themes. The badnews is that is not helping you. Sorry. I think we need to dig a little further. Based on the additional pages you mentioned (adding/editing tenant/application) I am wondering if something specific with the edits you have made to the themes. The fact that you change the order they are loaded in makes that less likely though. What are the sizes of the tpl folders for your themes?

david.cuen

@mark-robustelli The size of the tpl folders for each theme is around 352 KB - 404 KB.

I would like to mention that the hang occurs when launching the instance in a Linux environment with the Docker image.
If I launch a local instance on a Windows computer against the same PostgreSQL database, the hang problem does not occur.

Version: 1.62.1
Platform: Windows 11 10.0 amd64

Instances are launched with ENV var:
FUSIONAUTH_APP_MEMORY=1024M
DATABASE_CONNECTION_HEALTHCHECK_99PERCENT_MS=500

When consulting Prometheus metrics with /api/status after the instance has recovered from the hang, errors appear in the theme search

            "prime-mvc.[/api/theme/search].errors": {
                "count": 2,
                "fifteenMinuteRate": 0.001224500060165843,
                "fiveMinuteRate": 0.002437052376257949,
                "meanRate": 0.0012614192847508726,
                "oneMinuteRate": 0.003282810949244636
            },

Any idea how to find the error that occurs in the internal call from fusionauth admin app to its API?

david.cuen

@mark-robustelli I've launched the instance in the infrastructure enabling an agentlib to perform remote debugging of the JVM.

Debugging the theme search process, I see following exception produced in the RESTClient response of the API call

com.inversoft.rest.JSONException: Failed to parse the HTTP response as JSON. Actual HTTP response body:
Note: Output has been truncated to the first 1024 of 2560141 bytes.

{"themes":[{"data":{},"defaultMessages":"#\n# Date and Time formats\n#\ndate-format=M/d/yyyy\ndate-time-format=M/d/yyyy hh:mm a z\ndate-time-seconds-format=M/d/yyyy hh:mm:ss a z\n\n#\n# Text used on the page (inside the HTML). You can create new key-value pairs here and use them in the templates.\n#\naccess-denied=Access denied\naccount=Account\naction=Action\nadd-two-factor=Add two-factor\nadd-webauthn-passkey=Add passkey\nback-to-login=Return to Login\ncancel=Cancel\ncaptcha-google-branding=This site is protected by reCAPTCHA and the Google <a href=\"https://policies.google.com/privacy\" class=\"text-indigo-500 hover:text-indigo-700 font-medium focus:outline-none focus:underline\">Privacy Policy</a> and <a href=\"https://policies.google.com/terms\" class=\"text-indigo-500 hover:text-indigo-700 font-medium focus:outline-none focus:underline\">Terms of Service</a> apply.\ncreated=Created\ncustomize=Customize\nauthorized-not-registered=Registration is required to access this application and your account has no

Cause:

"com.fasterxml.jackson.databind.JsonMappingException: Premature EOF (through reference chain: io.fusionauth.domain.api.ThemeSearchResponse["themes"]->java.util.ArrayList[9])"

StackTrace:

com.inversoft.rest.JSONResponseHandler.apply(JSONResponseHandler.java:73)
com.inversoft.rest.RESTClient.go(RESTClient.java:430)
io.fusionauth.client.FusionAuthClient.searchThemes(FusionAuthClient.java:5372)
io.fusionauth.app.action.admin.theme.IndexAction.lambda$search$0(IndexAction.java:52)
io.fusionauth.client.LambdaDelegate.execute(LambdaDelegate.java:58)
io.fusionauth.app.action.admin.theme.IndexAction.search(IndexAction.java:52)
io.fusionauth.app.action.admin.BaseSearchAction.execute(BaseSearchAction.java:77)
java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103)
java.base/java.lang.reflect.Method.invoke(Method.java:580)
org.primeframework.mvc.util.ReflectionUtils.invoke(ReflectionUtils.java:443)
org.primeframework.mvc.action.DefaultActionInvocationWorkflow.execute(DefaultActionInvocationWorkflow.java:77)
org.primeframework.mvc.action.DefaultActionInvocationWorkflow.perform(DefaultActionInvocationWorkflow.java:60)
org.primeframework.mvc.workflow.SubWorkflowChain.continueWorkflow(SubWorkflowChain.java:50)
org.primeframework.mvc.message.DefaultMessageWorkflow.perform(DefaultMessageWorkflow.java:50)
org.primeframework.mvc.workflow.SubWorkflowChain.continueWorkflow(SubWorkflowChain.java:50)
org.primeframework.mvc.validation.DefaultValidationWorkflow.perform(DefaultValidationWorkflow.java:45)
org.primeframework.mvc.workflow.SubWorkflowChain.continueWorkflow(SubWorkflowChain.java:50)
org.primeframework.mvc.security.DefaultSecurityWorkflow.perform(DefaultSecurityWorkflow.java:79)
org.primeframework.mvc.workflow.SubWorkflowChain.continueWorkflow(SubWorkflowChain.java:50)
org.primeframework.mvc.parameter.DefaultPostParameterWorkflow.perform(DefaultPostParameterWorkflow.java:49)
org.primeframework.mvc.workflow.SubWorkflowChain.continueWorkflow(SubWorkflowChain.java:50)
org.primeframework.mvc.content.DefaultContentWorkflow.perform(DefaultContentWorkflow.java:74)
org.primeframework.mvc.workflow.SubWorkflowChain.continueWorkflow(SubWorkflowChain.java:50)
org.primeframework.mvc.parameter.DefaultParameterWorkflow.perform(DefaultParameterWorkflow.java:58)
org.primeframework.mvc.workflow.SubWorkflowChain.continueWorkflow(SubWorkflowChain.java:50)
org.primeframework.mvc.parameter.DefaultURIParameterWorkflow.perform(DefaultURIParameterWorkflow.java:92)
org.primeframework.mvc.workflow.SubWorkflowChain.continueWorkflow(SubWorkflowChain.java:50)
org.primeframework.mvc.scope.DefaultScopeRetrievalWorkflow.perform(DefaultScopeRetrievalWorkflow.java:50)
org.primeframework.mvc.workflow.SubWorkflowChain.continueWorkflow(SubWorkflowChain.java:50)
org.primeframework.mvc.action.DefaultActionMappingWorkflow.perform(DefaultActionMappingWorkflow.java:130)
org.primeframework.mvc.workflow.SubWorkflowChain.continueWorkflow(SubWorkflowChain.java:50)
org.primeframework.mvc.security.DefaultSavedRequestWorkflow.perform(DefaultSavedRequestWorkflow.java:65)
org.primeframework.mvc.workflow.SubWorkflowChain.continueWorkflow(SubWorkflowChain.java:50)
org.primeframework.mvc.cors.CORSFilter.doFilter(CORSFilter.java:188)
org.primeframework.mvc.cors.CORSRequestWorkflow.perform(CORSRequestWorkflow.java:66)
org.primeframework.mvc.workflow.SubWorkflowChain.continueWorkflow(SubWorkflowChain.java:50)
org.primeframework.mvc.workflow.DefaultMVCWorkflow.perform(DefaultMVCWorkflow.java:109)
org.primeframework.mvc.PrimeMVCRequestHandler.handle(PrimeMVCRequestHandler.java:76)
io.fusionauth.http.server.internal.HTTPWorker.run(HTTPWorker.java:183)
java.base/java.lang.VirtualThread.run(VirtualThread.java:329)

mark.robustelli

@david-cuen sorry for the delayed response. I was unexpectedly out. Thanks for trying to dig a bit deeper. The message you definitely leads me to believe that it is either a specific theme or as you said a size thing. Both are a bit confusing to me as the size you mentioned for your themes does not seem that bit and you stated that you tried loading the themes in different orders and that did not seem to make a difference. I will see if I can dig a little deeper on these error messages and see if there is anything I can find.

david.cuen

@mark-robustelli The error occurs when the response from the theme search is greater than 2.5 MB.

I don't know if it's related or not, but it happens to me in versions of FusionAuth later than 1.60.2.
In the release notes for version 1.61.0, an upgrade of the Jackson dependencies has just been made

https://fusionauth.io/docs/release-notes/#version-1-61-0

Upgrade com.fasterxml.jackson.* 2.17.2 -> 2.19.2

In that version of Jackson, the collection deserialization process has been modified.

mark.robustelli

@david-cuen I wasn't able to find much else out on Friday. Awesome that it looks like you have it narrowed down to the size. I will try to replicate this. I will let you know if I can.

mark.robustelli

@david-cuen This is really weird. I was not able to replicate it. There is one more thing I may try later if I get some time. That is to make one template that big and try that.

In the mean time, I went back to look at the error message. "Premature EOF" and the fact that it was working on a windows box has be back to believing that a funky character or something is causing the error on the linux box. I wish we could narrow it down. Have you tried to install into another linux environment, EC2 for example, and see if you get the same issue? (That may tell us if it is a linux thing or if it is your one specific environment.) Also, someone mentioned to ask if you have any proxies running in front of FusionAuth?

david.cuen

@mark-robustelli
The instance is currently running in an AWS EC2 Linux environment. I'm sorry, but I can't change the infrastructure.

The error occurs when processing the response to the internal call made from FusionAuth with the RESTClient to the endpoint
http://localhost:9012/api/theme/search.
As far as I know, there is no proxy running in front of this internal FusionAuth call.

I have tried making calls to /api/theme/search from outside of the instance container.
The result is always correct and with the complete JSON response.

Regarding funky characters in themes, in my tests they are all clones of the default advanced theme. So I can't understand the difference in the error when having +1 default advanced theme, only what was mentioned about the final size of the response.

I'm going to try to reproduce it using default simple theme clones.

mark.robustelli

@david-cuen Thanks for your patience and dedication to seeing this through. It would help a ton if you could find something reproducible. Let me know what you find and I can continue to try it on this end.