@hvfa Apologies for being a bit AWOL. I hope for things to slow down and be able to take a look at this a little more closely next week.

There are a couple different scenarios where a login record could have a blank application Id. Usually it is #1 or #2. It occurs in scenarios where the user can have a JWT/access token that does not have the application Id in it.

If a user is not registered for the Application they are logging into FusionAuth makes a login record when a user is created since FA makes a JWT upon user creation If you use the Login API, you can log in without an App ID because you don't have to provide an application on the API call.

@batmysta Going to link to this issue.

@batmysta I know this is a bit of a "trick" answer, but the reality is the right answer depends on what exactly you want the user to have access for. I understand that the roles in FusionAuth may give more access than you may like (I.E. MFA is a tenant level settings, but there is no role just for MFA edit.), but there are some other options.

Again, depending on what you want to do and what version you are running, there is the idea of the Tenant Manager applicaiton. This will still not help you with MFA settings thought.

The other option is using the APIs. Since everything in The FusionAuth admin UI is API first, you could create your own application that would allow users you choose to edit them.

Hope this is useful.

I think the way I'd approach this is:

import all users into FusionAuth

At cutover time:

look at local database to see which password hashes had changed pull the user data from FusionAuth for each of these users delete the user re-import the user with the new password hash and the FusionAuth data, maintaining the same userId (if you provide the UUID, we'll use that)

I get that is an additional complexity, but hopefully that helps.

This is totally possible.

You want to start by understanding FusionAuth passkey setup and the normal flow.

Then, in your application, probably using one of the client libraries, you want to do the following for a user:

see if a user has a passkey set up, using the "retrieve a passkey" API. If this returns 0 passkeys, show the prompt. for the prompt, you have two options: use the API/client library to start the passkey registration process from within your application directly send them to the user management page to add a passkey (requires a paid license)

The right way to do the latter depends on your application needs (are you okay with a redirect) and whether or not you have at least a starter license.

For reporting on the number of users that have set up passkeys, unfortunately you have to query all your users and then pull the passkey data individually. There's no way to use the elasticsearch syntax to do the query as of yet. There's an open github issue to add that functionality.

Hi @mark-robustelli

I’ve been investigating the issue over the last few weeks.

Having taken a JVM dump while the issue was occurring, here is the stack trace for the thread causing the deadlock

{ "tid": "8820", "name": "HTTP client [\/[0:0:0:0:0:0:0:1]:50154]", "stack": [ "java.base\/jdk.internal.misc.Unsafe.park(Native Method)", "java.base\/java.lang.VirtualThread.parkOnCarrierThread(VirtualThread.java:677)", "java.base\/java.lang.VirtualThread.parkNanos(VirtualThread.java:648)", "java.base\/java.lang.System$2.parkVirtualThread(System.java:2652)", "java.base\/jdk.internal.misc.VirtualThreads.park(VirtualThreads.java:67)", "java.base\/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:408)", "java.base\/sun.nio.ch.Poller.pollIndirect(Poller.java:137)", "java.base\/sun.nio.ch.Poller.poll(Poller.java:102)", "java.base\/sun.nio.ch.Poller.poll(Poller.java:87)", "java.base\/sun.nio.ch.NioSocketImpl.park(NioSocketImpl.java:175)", "java.base\/sun.nio.ch.NioSocketImpl.timedRead(NioSocketImpl.java:280)", "java.base\/sun.nio.ch.NioSocketImpl.implRead(NioSocketImpl.java:304)", "java.base\/sun.nio.ch.NioSocketImpl.read(NioSocketImpl.java:346)", "java.base\/sun.nio.ch.NioSocketImpl$1.read(NioSocketImpl.java:796)", "java.base\/java.net.Socket$SocketInputStream.read(Socket.java:1099)", "java.base\/java.io.BufferedInputStream.fill(BufferedInputStream.java:291)", "java.base\/java.io.BufferedInputStream.read1(BufferedInputStream.java:347)", "java.base\/java.io.BufferedInputStream.implRead(BufferedInputStream.java:420)", "java.base\/java.io.BufferedInputStream.read(BufferedInputStream.java:399)", "java.base\/sun.net.www.http.ChunkedInputStream.fastRead(ChunkedInputStream.java:244)", "java.base\/sun.net.www.http.ChunkedInputStream.read(ChunkedInputStream.java:698)", "java.base\/java.io.FilterInputStream.read(FilterInputStream.java:119)", "java.base\/sun.net.www.protocol.http.HttpURLConnection$HttpInputStream.read(HttpURLConnection.java:3677)", "java.base\/java.io.BufferedInputStream.read1(BufferedInputStream.java:345)", "java.base\/java.io.BufferedInputStream.implRead(BufferedInputStream.java:420)", "java.base\/java.io.BufferedInputStream.read(BufferedInputStream.java:405)", "com.inversoft.rest.JSONResponseHandler$BetterBufferedInputStream.read(JSONResponseHandler.java:127)", "com.fasterxml.jackson.core.json.UTF8StreamJsonParser._loadMore(UTF8StreamJsonParser.java:220)", "com.fasterxml.jackson.core.json.UTF8StreamJsonParser._loadMoreGuaranteed(UTF8StreamJsonParser.java:2457)", "com.fasterxml.jackson.core.json.UTF8StreamJsonParser._finishString2(UTF8StreamJsonParser.java:2540)", "com.fasterxml.jackson.core.json.UTF8StreamJsonParser._finishAndReturnString(UTF8StreamJsonParser.java:2520)", "com.fasterxml.jackson.core.json.UTF8StreamJsonParser.getText(UTF8StreamJsonParser.java:294)", "com.fasterxml.jackson.databind.deser.std.StringDeserializer.deserialize(StringDeserializer.java:42)", "com.fasterxml.jackson.databind.deser.std.StringDeserializer.deserialize(StringDeserializer.java:11)", "com.fasterxml.jackson.databind.deser.impl.FieldProperty.deserializeAndSet(FieldProperty.java:137)", "com.fasterxml.jackson.databind.deser.BeanDeserializer.vanillaDeserialize(BeanDeserializer.java:302)", "com.fasterxml.jackson.databind.deser.BeanDeserializer.deserialize(BeanDeserializer.java:169)", "com.fasterxml.jackson.databind.deser.impl.FieldProperty.deserializeAndSet(FieldProperty.java:137)", "com.fasterxml.jackson.databind.deser.BeanDeserializer.vanillaDeserialize(BeanDeserializer.java:302)", "com.fasterxml.jackson.databind.deser.BeanDeserializer.deserialize(BeanDeserializer.java:169)", "com.fasterxml.jackson.databind.deser.std.CollectionDeserializer._deserializeNoNullChecks(CollectionDeserializer.java:501)", "com.fasterxml.jackson.databind.deser.std.CollectionDeserializer._deserializeFromArray(CollectionDeserializer.java:358)", "com.fasterxml.jackson.databind.deser.std.CollectionDeserializer.deserialize(CollectionDeserializer.java:245)", "com.fasterxml.jackson.databind.deser.std.CollectionDeserializer.deserialize(CollectionDeserializer.java:29)", "com.fasterxml.jackson.databind.deser.impl.FieldProperty.deserializeAndSet(FieldProperty.java:137)", "com.fasterxml.jackson.databind.deser.BeanDeserializer.vanillaDeserialize(BeanDeserializer.java:302)", "com.fasterxml.jackson.databind.deser.BeanDeserializer.deserialize(BeanDeserializer.java:169)", "com.fasterxml.jackson.databind.deser.DefaultDeserializationContext.readRootValue(DefaultDeserializationContext.java:342)", "com.fasterxml.jackson.databind.ObjectMapper._readMapAndClose(ObjectMapper.java:4971)", "com.fasterxml.jackson.databind.ObjectMapper.readValue(ObjectMapper.java:3924)", "com.inversoft.rest.JSONResponseHandler.apply(JSONResponseHandler.java:68)", "com.inversoft.rest.RESTClient.go(RESTClient.java:430)", "io.fusionauth.client.FusionAuthClient.searchThemes(FusionAuthClient.java:5372)", "io.fusionauth.app.action.admin.theme.IndexAction.lambda$search$0(IndexAction.java:52)", "io.fusionauth.client.LambdaDelegate.execute(LambdaDelegate.java:58)", "io.fusionauth.app.action.admin.theme.IndexAction.search(IndexAction.java:52)", "io.fusionauth.app.action.admin.BaseSearchAction.execute(BaseSearchAction.java:77)", "java.base\/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103)", "java.base\/java.lang.reflect.Method.invoke(Method.java:580)", "org.primeframework.mvc.util.ReflectionUtils.invoke(ReflectionUtils.java:443)", "org.primeframework.mvc.action.DefaultActionInvocationWorkflow.execute(DefaultActionInvocationWorkflow.java:77)", "org.primeframework.mvc.action.DefaultActionInvocationWorkflow.perform(DefaultActionInvocationWorkflow.java:60)", "org.primeframework.mvc.workflow.SubWorkflowChain.continueWorkflow(SubWorkflowChain.java:50)", "org.primeframework.mvc.message.DefaultMessageWorkflow.perform(DefaultMessageWorkflow.java:50)", "org.primeframework.mvc.workflow.SubWorkflowChain.continueWorkflow(SubWorkflowChain.java:50)", "org.primeframework.mvc.validation.DefaultValidationWorkflow.perform(DefaultValidationWorkflow.java:45)", "org.primeframework.mvc.workflow.SubWorkflowChain.continueWorkflow(SubWorkflowChain.java:50)", "org.primeframework.mvc.security.DefaultSecurityWorkflow.perform(DefaultSecurityWorkflow.java:79)", "org.primeframework.mvc.workflow.SubWorkflowChain.continueWorkflow(SubWorkflowChain.java:50)", "org.primeframework.mvc.parameter.DefaultPostParameterWorkflow.perform(DefaultPostParameterWorkflow.java:49)", "org.primeframework.mvc.workflow.SubWorkflowChain.continueWorkflow(SubWorkflowChain.java:50)", "org.primeframework.mvc.content.DefaultContentWorkflow.perform(DefaultContentWorkflow.java:74)", "org.primeframework.mvc.workflow.SubWorkflowChain.continueWorkflow(SubWorkflowChain.java:50)", "org.primeframework.mvc.parameter.DefaultParameterWorkflow.perform(DefaultParameterWorkflow.java:58)", "org.primeframework.mvc.workflow.SubWorkflowChain.continueWorkflow(SubWorkflowChain.java:50)", "org.primeframework.mvc.parameter.DefaultURIParameterWorkflow.perform(DefaultURIParameterWorkflow.java:92)", "org.primeframework.mvc.workflow.SubWorkflowChain.continueWorkflow(SubWorkflowChain.java:50)", "org.primeframework.mvc.scope.DefaultScopeRetrievalWorkflow.perform(DefaultScopeRetrievalWorkflow.java:50)", "org.primeframework.mvc.workflow.SubWorkflowChain.continueWorkflow(SubWorkflowChain.java:50)", "org.primeframework.mvc.action.DefaultActionMappingWorkflow.perform(DefaultActionMappingWorkflow.java:130)", "org.primeframework.mvc.workflow.SubWorkflowChain.continueWorkflow(SubWorkflowChain.java:50)", "org.primeframework.mvc.security.DefaultSavedRequestWorkflow.perform(DefaultSavedRequestWorkflow.java:65)", "org.primeframework.mvc.workflow.SubWorkflowChain.continueWorkflow(SubWorkflowChain.java:50)", "org.primeframework.mvc.cors.CORSFilter.doFilter(CORSFilter.java:188)", "org.primeframework.mvc.cors.CORSRequestWorkflow.perform(CORSRequestWorkflow.java:66)", "org.primeframework.mvc.workflow.SubWorkflowChain.continueWorkflow(SubWorkflowChain.java:50)", "org.primeframework.mvc.workflow.DefaultMVCWorkflow.perform(DefaultMVCWorkflow.java:109)", "org.primeframework.mvc.PrimeMVCRequestHandler.handle(PrimeMVCRequestHandler.java:76)", "io.fusionauth.http.server.internal.HTTPWorker.run(HTTPWorker.java:183)", "java.base\/java.lang.VirtualThread.run(VirtualThread.java:329)" ] }

I enabled trace logging in io.fusionauth.http.server.internal to analyse the process in HTTPWorker

2026-03-12 03:53:18.758 PM TRACE io.fusionauth.http.server.internal.HTTPWorker - [993] Running HTTP worker. Block while we wait to read the preamble 2026-03-12 03:53:18.758 PM TRACE io.fusionauth.http.server.internal.HTTPWorker - [993] Set state [Process]. Call the request handler. 2026-03-12 03:53:18.773 PM TRACE io.fusionauth.http.server.internal.HTTPWorker - [993] Handler completed successfully 2026-03-12 03:53:18.773 PM TRACE io.fusionauth.http.server.internal.HTTPWorker - [993] Enter Keep-Alive state [KeepAlive] Reset socket timeout [60000]. 2026-03-12 03:53:18.773 PM TRACE io.fusionauth.http.server.internal.HTTPWorker - [993] Running HTTP worker. Block while we wait to read the preamble 2026-03-12 03:53:18.775 PM TRACE io.fusionauth.http.server.internal.HTTPWorker - [993] Set state [Process]. Call the request handler. 2026-03-12 03:53:18.776 PM TRACE io.fusionauth.http.server.internal.HTTPWorker - [993] Handler completed successfully 2026-03-12 03:53:18.776 PM TRACE io.fusionauth.http.server.internal.HTTPWorker - [993] Enter Keep-Alive state [KeepAlive] Reset socket timeout [60000]. 2026-03-12 03:53:18.776 PM TRACE io.fusionauth.http.server.internal.HTTPWorker - [993] Running HTTP worker. Block while we wait to read the preamble 2026-03-12 03:53:18.777 PM TRACE io.fusionauth.http.server.internal.HTTPWorker - [993] Set state [Process]. Call the request handler. 2026-03-12 03:53:19.977 PM TRACE io.fusionauth.http.server.internal.HTTPServerThread - [993] Check worker in state [Write] 2026-03-12 03:53:21.978 PM TRACE io.fusionauth.http.server.internal.HTTPServerThread - [993] Check worker in state [Write] 2026-03-12 03:53:23.979 PM TRACE io.fusionauth.http.server.internal.HTTPServerThread - [993] Check worker in state [Write] ... ... 2026-03-12 03:55:54.019 PM TRACE io.fusionauth.http.server.internal.HTTPServerThread - [993] Check worker in state [Write] 2026-03-12 03:55:56.019 PM DEBUG io.fusionauth.http.server.internal.HTTPServerThread - [993] Check worker in state [Write] writingSlow=[true] writeThroughput=[16271] minimumWriteThroughput=[16384] 2026-03-12 03:55:56.019 PM DEBUG io.fusionauth.http.server.internal.HTTPServerThread - [993] Closing connection readingSlow=[false] writingSlow=[true] timedOut=[false] Min write throughput [16384], actual throughput [16271]. 2026-03-12 03:55:56.019 PM DEBUG io.fusionauth.http.server.internal.HTTPServerThread - [993] Closing client connection [/127.0.0.1:36278] due to inactivity 2026-03-12 03:55:56.021 PM TRACE io.fusionauth.http.server.internal.HTTPServerThread - Thread dump from server side. 2026-03-12 03:55:56.024 PM DEBUG io.fusionauth.http.server.internal.HTTPWorker - [993] Closing socket. The socket was closed by a client, proxy or otherwise.

The HTTPWorker begins the process of reading and writing the response for the request, and continues for 2.6 minutes until the HTTPServerThread terminates it because the minimumWriteThroughput (16KB/sec) threshold has not been met.

HTTPServerCleanerThread kills the server-side connection mid-response due to a write throughput check that measures average bytes/sec since the first socket write.
The client is waiting for data that the server is generating very slowly.

Analyzing why the server is writing so slowly (16271 bytes/sec):
The call is http://localhost:9012 — FusionAuth calls itself. Client and server virtual threads share the same JVM's carrier thread pool.
With Kubernetes CPU limit = 1000m, JDK 21 uses UseContainerSupport (default), so availableProcessors() = 1 → only 1 carrier thread.

com.inversoft.rest.JSONResponseHandler$BetterBufferedInputStream.read(byte[], int, int) is synchronized — causes virtual thread carrier thread pinning in JDK 21.

Client virtual thread enters synchronized read() → calls ChunkedInputStream.fastRead() → calls socket.read() which blocks → PINS the carrier thread (can't unmount from synchronized block) Server virtual thread needs to write more data but cannot get a carrier thread (carrier thread pinned by client) Neither makes progress → throughput decays over time as numberOfBytesWritten is fixed but elapsed keeps growing

BetterBufferedInputStream.read() pins that single carrier thread while blocking on ChunkedInputStream.fastRead(), leaving no carrier thread for the server to write more data. The resulting ping-pong limits throughput to ~16 KB/sec on a 2.5MB response.

HTTPServerCleanerThread computes average throughput since the very first write (not a recent window). With 2.5MB written at 16271 bytes/sec average, that's ~156 seconds (2.6 min) before the average decays below the 16384 bytes/sec threshold.

With 9 themes, the response is small enough to fit entirely within the kernel’s socket buffer (~128KB). The server writes the entire content in one go without blocking on socket.write() (the buffer does not fill up), transitions to the KeepAlive state, and the cleaner can no longer terminate the connection due to write throughput.

As a workaround, I resolved the issue increasing the Kubernetes CPU limit from 1000m to 1050m.
Kubernetes translates the CPU limit into the container’s cgroups, setting 2 carrier threads instead of 1.

With that CPU limit, all themes are recovered properly, also with 35 themes (~6.2 MB response).

Can you reproduce it setting Kubernetes CPU limit to 1000m (1 core) in your environment?