@elliotdickison Can you let us know what third party OIDC IdP you are working with?

@elliotdickison Thanks for the feedback.

I'd love to chat a bit more to understand the problem.

Will send you an email.

Because advanced themes are so customizable, they can be hard to upgrade. Here's some ways to make it easier.

When you create a new theme, start from the default version. Commit it to git before you change anything. Use the FusionAuth CLI to download/upload your theme during development and CI/CD. When a new theme comes out, clone or pull the latest from the theme history repo. Run this command to see what has changed: git format-patch 1.61.0..1.64.1 --stdout > update-themes.patch (this shows the changes between 1.61.0 and 1.64.1; adjust as needed for your installed version and the target version). Go to your theme git repo and apply the changes: git am --3way update-themes.patch which will attempt to automatically merge the changes. If there are conflicts, you can resolve them manually and then run git am --continue.

You can also use a 3 way diffing tool like diff3 or kdiff3 to visualize the changes.

These upgrade notes also provide detailed human friendly instructions on the changes.

There are a couple different scenarios where a login record could have a blank application Id. Usually it is #1 or #2. It occurs in scenarios where the user can have a JWT/access token that does not have the application Id in it.

If a user is not registered for the Application they are logging into FusionAuth makes a login record when a user is created since FA makes a JWT upon user creation If you use the Login API, you can log in without an App ID because you don't have to provide an application on the API call.

@batmysta Going to link to this issue.

@batmysta I know this is a bit of a "trick" answer, but the reality is the right answer depends on what exactly you want the user to have access for. I understand that the roles in FusionAuth may give more access than you may like (I.E. MFA is a tenant level settings, but there is no role just for MFA edit.), but there are some other options.

Again, depending on what you want to do and what version you are running, there is the idea of the Tenant Manager applicaiton. This will still not help you with MFA settings thought.

The other option is using the APIs. Since everything in The FusionAuth admin UI is API first, you could create your own application that would allow users you choose to edit them.

Hope this is useful.

@hvfa Apologies for being a bit AWOL. I hope for things to slow down and be able to take a look at this a little more closely next week.

I think the way I'd approach this is:

import all users into FusionAuth

At cutover time:

look at local database to see which password hashes had changed pull the user data from FusionAuth for each of these users delete the user re-import the user with the new password hash and the FusionAuth data, maintaining the same userId (if you provide the UUID, we'll use that)

I get that is an additional complexity, but hopefully that helps.

It's so easy to mix up those IDs when you're moving between FusionAuth and the Google console! It’s definitely one of those things that’s right under your nose but impossible to see until someone points it out. Glad to hear you got the callback working - that 'invalid_client' error can be a real headache when everything else looks correct.

I've found that setting the mobile number as the loginId is the most reliable way to handle this right now. You can just tweak the theme labels to say 'Mobile Number' instead of 'Username' so it's clear to the users. It’s a bit of a manual setup for the SMS verification part via the API, but it gets the job done without waiting for a native feature update.

@mark-robustelli cheers, that was a useful post. The IAMShowcase tools did help me filter out what was correct and what was not. Eventually I found that the compression config settings on the passed request were not correct.

Managing users in bulk can definitely be a bit nerve-wracking when you're doing it for the first time.
If you’re looking for a quick way to handle this, the FusionAuth Search API is probably your best friend here. You can run a query to get the IDs of the users you want to target, and then loop through them with a simple script using the Delete User API.
If you just want to "deactivate" them instead of a hard delete, you can toggle the active flag to false in the User object. This is usually a safer bet if you think you might need to reactivate them later without losing all their historical data or linked identities. Just a heads-up: make sure you have a good backup of your database before running any bulk scripts—it's saved my skin more than once!

Webhook errors can be a real pain to debug since they often fail silently or with very generic messages. I’ve found that most of the time it comes down to either a TLS/SSL handshake issue or the endpoint expecting a specific header that FusionAuth isn't sending.
One thing that really helps is using a tool like Webhook.site or RequestBin just to see if the event is actually firing and what the payload looks like. If it works there but not on your server, it’s almost certainly a firewall or certificate trust issue on your end. Also, double-check that your secret is matching up perfectly - it’s easy for a stray whitespace to throw the whole signature validation off!

I ran into a similar issue recently while trying to upgrade my stack. It seems like FusionAuth is generally pretty stable on the LTS versions, but things can get a bit 'interesting' if you're jumping to the absolute latest bleeding-edge Node release before they've officially validated it.
I've found that sticking to the even-numbered LTS releases usually saves a lot of headache with the client libraries. Are you seeing specific crash logs, or is it just a dependency resolution error during the build?

@mark-robustelli I just upgraded to 1.64, still happens, exact same behavior

Hi @mark-robustelli

I’ve been investigating the issue over the last few weeks.

Having taken a JVM dump while the issue was occurring, here is the stack trace for the thread causing the deadlock

{ "tid": "8820", "name": "HTTP client [\/[0:0:0:0:0:0:0:1]:50154]", "stack": [ "java.base\/jdk.internal.misc.Unsafe.park(Native Method)", "java.base\/java.lang.VirtualThread.parkOnCarrierThread(VirtualThread.java:677)", "java.base\/java.lang.VirtualThread.parkNanos(VirtualThread.java:648)", "java.base\/java.lang.System$2.parkVirtualThread(System.java:2652)", "java.base\/jdk.internal.misc.VirtualThreads.park(VirtualThreads.java:67)", "java.base\/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:408)", "java.base\/sun.nio.ch.Poller.pollIndirect(Poller.java:137)", "java.base\/sun.nio.ch.Poller.poll(Poller.java:102)", "java.base\/sun.nio.ch.Poller.poll(Poller.java:87)", "java.base\/sun.nio.ch.NioSocketImpl.park(NioSocketImpl.java:175)", "java.base\/sun.nio.ch.NioSocketImpl.timedRead(NioSocketImpl.java:280)", "java.base\/sun.nio.ch.NioSocketImpl.implRead(NioSocketImpl.java:304)", "java.base\/sun.nio.ch.NioSocketImpl.read(NioSocketImpl.java:346)", "java.base\/sun.nio.ch.NioSocketImpl$1.read(NioSocketImpl.java:796)", "java.base\/java.net.Socket$SocketInputStream.read(Socket.java:1099)", "java.base\/java.io.BufferedInputStream.fill(BufferedInputStream.java:291)", "java.base\/java.io.BufferedInputStream.read1(BufferedInputStream.java:347)", "java.base\/java.io.BufferedInputStream.implRead(BufferedInputStream.java:420)", "java.base\/java.io.BufferedInputStream.read(BufferedInputStream.java:399)", "java.base\/sun.net.www.http.ChunkedInputStream.fastRead(ChunkedInputStream.java:244)", "java.base\/sun.net.www.http.ChunkedInputStream.read(ChunkedInputStream.java:698)", "java.base\/java.io.FilterInputStream.read(FilterInputStream.java:119)", "java.base\/sun.net.www.protocol.http.HttpURLConnection$HttpInputStream.read(HttpURLConnection.java:3677)", "java.base\/java.io.BufferedInputStream.read1(BufferedInputStream.java:345)", "java.base\/java.io.BufferedInputStream.implRead(BufferedInputStream.java:420)", "java.base\/java.io.BufferedInputStream.read(BufferedInputStream.java:405)", "com.inversoft.rest.JSONResponseHandler$BetterBufferedInputStream.read(JSONResponseHandler.java:127)", "com.fasterxml.jackson.core.json.UTF8StreamJsonParser._loadMore(UTF8StreamJsonParser.java:220)", "com.fasterxml.jackson.core.json.UTF8StreamJsonParser._loadMoreGuaranteed(UTF8StreamJsonParser.java:2457)", "com.fasterxml.jackson.core.json.UTF8StreamJsonParser._finishString2(UTF8StreamJsonParser.java:2540)", "com.fasterxml.jackson.core.json.UTF8StreamJsonParser._finishAndReturnString(UTF8StreamJsonParser.java:2520)", "com.fasterxml.jackson.core.json.UTF8StreamJsonParser.getText(UTF8StreamJsonParser.java:294)", "com.fasterxml.jackson.databind.deser.std.StringDeserializer.deserialize(StringDeserializer.java:42)", "com.fasterxml.jackson.databind.deser.std.StringDeserializer.deserialize(StringDeserializer.java:11)", "com.fasterxml.jackson.databind.deser.impl.FieldProperty.deserializeAndSet(FieldProperty.java:137)", "com.fasterxml.jackson.databind.deser.BeanDeserializer.vanillaDeserialize(BeanDeserializer.java:302)", "com.fasterxml.jackson.databind.deser.BeanDeserializer.deserialize(BeanDeserializer.java:169)", "com.fasterxml.jackson.databind.deser.impl.FieldProperty.deserializeAndSet(FieldProperty.java:137)", "com.fasterxml.jackson.databind.deser.BeanDeserializer.vanillaDeserialize(BeanDeserializer.java:302)", "com.fasterxml.jackson.databind.deser.BeanDeserializer.deserialize(BeanDeserializer.java:169)", "com.fasterxml.jackson.databind.deser.std.CollectionDeserializer._deserializeNoNullChecks(CollectionDeserializer.java:501)", "com.fasterxml.jackson.databind.deser.std.CollectionDeserializer._deserializeFromArray(CollectionDeserializer.java:358)", "com.fasterxml.jackson.databind.deser.std.CollectionDeserializer.deserialize(CollectionDeserializer.java:245)", "com.fasterxml.jackson.databind.deser.std.CollectionDeserializer.deserialize(CollectionDeserializer.java:29)", "com.fasterxml.jackson.databind.deser.impl.FieldProperty.deserializeAndSet(FieldProperty.java:137)", "com.fasterxml.jackson.databind.deser.BeanDeserializer.vanillaDeserialize(BeanDeserializer.java:302)", "com.fasterxml.jackson.databind.deser.BeanDeserializer.deserialize(BeanDeserializer.java:169)", "com.fasterxml.jackson.databind.deser.DefaultDeserializationContext.readRootValue(DefaultDeserializationContext.java:342)", "com.fasterxml.jackson.databind.ObjectMapper._readMapAndClose(ObjectMapper.java:4971)", "com.fasterxml.jackson.databind.ObjectMapper.readValue(ObjectMapper.java:3924)", "com.inversoft.rest.JSONResponseHandler.apply(JSONResponseHandler.java:68)", "com.inversoft.rest.RESTClient.go(RESTClient.java:430)", "io.fusionauth.client.FusionAuthClient.searchThemes(FusionAuthClient.java:5372)", "io.fusionauth.app.action.admin.theme.IndexAction.lambda$search$0(IndexAction.java:52)", "io.fusionauth.client.LambdaDelegate.execute(LambdaDelegate.java:58)", "io.fusionauth.app.action.admin.theme.IndexAction.search(IndexAction.java:52)", "io.fusionauth.app.action.admin.BaseSearchAction.execute(BaseSearchAction.java:77)", "java.base\/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103)", "java.base\/java.lang.reflect.Method.invoke(Method.java:580)", "org.primeframework.mvc.util.ReflectionUtils.invoke(ReflectionUtils.java:443)", "org.primeframework.mvc.action.DefaultActionInvocationWorkflow.execute(DefaultActionInvocationWorkflow.java:77)", "org.primeframework.mvc.action.DefaultActionInvocationWorkflow.perform(DefaultActionInvocationWorkflow.java:60)", "org.primeframework.mvc.workflow.SubWorkflowChain.continueWorkflow(SubWorkflowChain.java:50)", "org.primeframework.mvc.message.DefaultMessageWorkflow.perform(DefaultMessageWorkflow.java:50)", "org.primeframework.mvc.workflow.SubWorkflowChain.continueWorkflow(SubWorkflowChain.java:50)", "org.primeframework.mvc.validation.DefaultValidationWorkflow.perform(DefaultValidationWorkflow.java:45)", "org.primeframework.mvc.workflow.SubWorkflowChain.continueWorkflow(SubWorkflowChain.java:50)", "org.primeframework.mvc.security.DefaultSecurityWorkflow.perform(DefaultSecurityWorkflow.java:79)", "org.primeframework.mvc.workflow.SubWorkflowChain.continueWorkflow(SubWorkflowChain.java:50)", "org.primeframework.mvc.parameter.DefaultPostParameterWorkflow.perform(DefaultPostParameterWorkflow.java:49)", "org.primeframework.mvc.workflow.SubWorkflowChain.continueWorkflow(SubWorkflowChain.java:50)", "org.primeframework.mvc.content.DefaultContentWorkflow.perform(DefaultContentWorkflow.java:74)", "org.primeframework.mvc.workflow.SubWorkflowChain.continueWorkflow(SubWorkflowChain.java:50)", "org.primeframework.mvc.parameter.DefaultParameterWorkflow.perform(DefaultParameterWorkflow.java:58)", "org.primeframework.mvc.workflow.SubWorkflowChain.continueWorkflow(SubWorkflowChain.java:50)", "org.primeframework.mvc.parameter.DefaultURIParameterWorkflow.perform(DefaultURIParameterWorkflow.java:92)", "org.primeframework.mvc.workflow.SubWorkflowChain.continueWorkflow(SubWorkflowChain.java:50)", "org.primeframework.mvc.scope.DefaultScopeRetrievalWorkflow.perform(DefaultScopeRetrievalWorkflow.java:50)", "org.primeframework.mvc.workflow.SubWorkflowChain.continueWorkflow(SubWorkflowChain.java:50)", "org.primeframework.mvc.action.DefaultActionMappingWorkflow.perform(DefaultActionMappingWorkflow.java:130)", "org.primeframework.mvc.workflow.SubWorkflowChain.continueWorkflow(SubWorkflowChain.java:50)", "org.primeframework.mvc.security.DefaultSavedRequestWorkflow.perform(DefaultSavedRequestWorkflow.java:65)", "org.primeframework.mvc.workflow.SubWorkflowChain.continueWorkflow(SubWorkflowChain.java:50)", "org.primeframework.mvc.cors.CORSFilter.doFilter(CORSFilter.java:188)", "org.primeframework.mvc.cors.CORSRequestWorkflow.perform(CORSRequestWorkflow.java:66)", "org.primeframework.mvc.workflow.SubWorkflowChain.continueWorkflow(SubWorkflowChain.java:50)", "org.primeframework.mvc.workflow.DefaultMVCWorkflow.perform(DefaultMVCWorkflow.java:109)", "org.primeframework.mvc.PrimeMVCRequestHandler.handle(PrimeMVCRequestHandler.java:76)", "io.fusionauth.http.server.internal.HTTPWorker.run(HTTPWorker.java:183)", "java.base\/java.lang.VirtualThread.run(VirtualThread.java:329)" ] }

I enabled trace logging in io.fusionauth.http.server.internal to analyse the process in HTTPWorker

2026-03-12 03:53:18.758 PM TRACE io.fusionauth.http.server.internal.HTTPWorker - [993] Running HTTP worker. Block while we wait to read the preamble 2026-03-12 03:53:18.758 PM TRACE io.fusionauth.http.server.internal.HTTPWorker - [993] Set state [Process]. Call the request handler. 2026-03-12 03:53:18.773 PM TRACE io.fusionauth.http.server.internal.HTTPWorker - [993] Handler completed successfully 2026-03-12 03:53:18.773 PM TRACE io.fusionauth.http.server.internal.HTTPWorker - [993] Enter Keep-Alive state [KeepAlive] Reset socket timeout [60000]. 2026-03-12 03:53:18.773 PM TRACE io.fusionauth.http.server.internal.HTTPWorker - [993] Running HTTP worker. Block while we wait to read the preamble 2026-03-12 03:53:18.775 PM TRACE io.fusionauth.http.server.internal.HTTPWorker - [993] Set state [Process]. Call the request handler. 2026-03-12 03:53:18.776 PM TRACE io.fusionauth.http.server.internal.HTTPWorker - [993] Handler completed successfully 2026-03-12 03:53:18.776 PM TRACE io.fusionauth.http.server.internal.HTTPWorker - [993] Enter Keep-Alive state [KeepAlive] Reset socket timeout [60000]. 2026-03-12 03:53:18.776 PM TRACE io.fusionauth.http.server.internal.HTTPWorker - [993] Running HTTP worker. Block while we wait to read the preamble 2026-03-12 03:53:18.777 PM TRACE io.fusionauth.http.server.internal.HTTPWorker - [993] Set state [Process]. Call the request handler. 2026-03-12 03:53:19.977 PM TRACE io.fusionauth.http.server.internal.HTTPServerThread - [993] Check worker in state [Write] 2026-03-12 03:53:21.978 PM TRACE io.fusionauth.http.server.internal.HTTPServerThread - [993] Check worker in state [Write] 2026-03-12 03:53:23.979 PM TRACE io.fusionauth.http.server.internal.HTTPServerThread - [993] Check worker in state [Write] ... ... 2026-03-12 03:55:54.019 PM TRACE io.fusionauth.http.server.internal.HTTPServerThread - [993] Check worker in state [Write] 2026-03-12 03:55:56.019 PM DEBUG io.fusionauth.http.server.internal.HTTPServerThread - [993] Check worker in state [Write] writingSlow=[true] writeThroughput=[16271] minimumWriteThroughput=[16384] 2026-03-12 03:55:56.019 PM DEBUG io.fusionauth.http.server.internal.HTTPServerThread - [993] Closing connection readingSlow=[false] writingSlow=[true] timedOut=[false] Min write throughput [16384], actual throughput [16271]. 2026-03-12 03:55:56.019 PM DEBUG io.fusionauth.http.server.internal.HTTPServerThread - [993] Closing client connection [/127.0.0.1:36278] due to inactivity 2026-03-12 03:55:56.021 PM TRACE io.fusionauth.http.server.internal.HTTPServerThread - Thread dump from server side. 2026-03-12 03:55:56.024 PM DEBUG io.fusionauth.http.server.internal.HTTPWorker - [993] Closing socket. The socket was closed by a client, proxy or otherwise.

The HTTPWorker begins the process of reading and writing the response for the request, and continues for 2.6 minutes until the HTTPServerThread terminates it because the minimumWriteThroughput (16KB/sec) threshold has not been met.

HTTPServerCleanerThread kills the server-side connection mid-response due to a write throughput check that measures average bytes/sec since the first socket write.
The client is waiting for data that the server is generating very slowly.

Analyzing why the server is writing so slowly (16271 bytes/sec):
The call is http://localhost:9012 — FusionAuth calls itself. Client and server virtual threads share the same JVM's carrier thread pool.
With Kubernetes CPU limit = 1000m, JDK 21 uses UseContainerSupport (default), so availableProcessors() = 1 → only 1 carrier thread.

com.inversoft.rest.JSONResponseHandler$BetterBufferedInputStream.read(byte[], int, int) is synchronized — causes virtual thread carrier thread pinning in JDK 21.

Client virtual thread enters synchronized read() → calls ChunkedInputStream.fastRead() → calls socket.read() which blocks → PINS the carrier thread (can't unmount from synchronized block) Server virtual thread needs to write more data but cannot get a carrier thread (carrier thread pinned by client) Neither makes progress → throughput decays over time as numberOfBytesWritten is fixed but elapsed keeps growing

BetterBufferedInputStream.read() pins that single carrier thread while blocking on ChunkedInputStream.fastRead(), leaving no carrier thread for the server to write more data. The resulting ping-pong limits throughput to ~16 KB/sec on a 2.5MB response.

HTTPServerCleanerThread computes average throughput since the very first write (not a recent window). With 2.5MB written at 16271 bytes/sec average, that's ~156 seconds (2.6 min) before the average decays below the 16384 bytes/sec threshold.

With 9 themes, the response is small enough to fit entirely within the kernel’s socket buffer (~128KB). The server writes the entire content in one go without blocking on socket.write() (the buffer does not fill up), transitions to the KeepAlive state, and the cleaner can no longer terminate the connection due to write throughput.

As a workaround, I resolved the issue increasing the Kubernetes CPU limit from 1000m to 1050m.
Kubernetes translates the CPU limit into the container’s cgroups, setting 2 carrier threads instead of 1.

With that CPU limit, all themes are recovered properly, also with 35 themes (~6.2 MB response).

Can you reproduce it setting Kubernetes CPU limit to 1000m (1 core) in your environment?