yves

yves

I've got a question regarding the reindexing of the ElasticSearch fusionauth_user index.

When we perform a reindex it seems that some users/documents get "lost".

Some observations:

We've enabled debugging on ElasticSearch and don't see any errors being thrown when the index is dropped and recreated.
The query that fails afterwards is a "nested" search for a particular registration users have.
When we PATCH or lock/unlock users then they appear in the ElasticSearch index again and the query succeeds.

What happens when the reindex is performed? I'm struggling to understand why or how this could happen because I assumed all users/documents are inserted in the new index again?

Right now our workaround is to PATCH all users so that we can be sure that they're in the index.

Any ideas? Pointers?

yves

@mark-robustelli Sure!

I do the following (I use HTTPie)

First I hit the authorize endpoint: http "https://myurl/oauth2/authorize?client_id=<myappid>&idp_hint=<myidpid>&nonce=<random generated nonce>&redirect_uri=/admin/login&response_type=code"
That gives me a Location which I follow: http "https://myurl/oauth2/redirect?client_id=<myappid>&identityProviderId=<myidpid>&state=<state>&nonce=<random generated nonce>"
Now again in the Location I get redirected to my IdP, I open that link in the browser and go through the flow of my IdP, once successfully authenticated it redirects me back to https://myurl/admin/login (the initial redirect URI) and I get the error I pasted earlier

Does this look correct to you?

yves

Thank you for the pointer @mark-robustelli .

However, also with this setting, I still get the following error message:

What am I doing wrong? Is this not referring to PKCE? Is this referring to something else? Where could I get further debug info? (Unfortunately the event log is empty.)

yves

Has anyone had a similar issue? Would really appreciate a bit of insights here. Ideally I'd like to force FusionAuth to stop using PKCE with the identity provider I configured.

yves

I've got an OpenID Connect identity provider which doesn't support PKCE.

I've read in the documentation that FusionAuth supports this and automatically falls back:

Proof Key for Code Exchange, more commonly referred to as PKCE (pronounced pixy) is an extension to the Authorization Code grant. This extension is intended to help secure the code exchange workflow utilized by this OpenID Connect configuration.
This extension is used by default on all OpenID Connect IdP configurations, and it cannot be disabled. The use of this extension is backwards compatible with identity providers that either do not require or support PKCE.
FusionAuth will pass along the required PKCE request parameters to the OpenID Connect identity provider and if the provider supports PKCE, the extension will be utilized, and if it is not supported it will be ignored.

From: https://fusionauth.io/docs/lifecycle/authenticate-users/identity-providers/overview-oidc

However when I use this my login always fails with "The code_verifier could not be determined", which suggests to me that it attempts to use PKCE.

How can I (just for testing) force disabling PKCE?

How does FusionAuth detect if an identity provider supports PKCE?

How can I further debug this?

yves

@dan Thanks for this!

I guess I'm struggling to understand how FusionAuth internally saves the user data. Is this unstructured (e.g. JSON) inside the relational database?

I guess I can't give this more structure, e.g. defining data types, and so on?

I guess there's also no "direct" access to the user data via an API? At least I didn't find anything.

yves

Thanks for the reply @dan!

Will answer inline.

@dan said in ElasticSearch reindex doesn't reindex all documents:

Hmmm. I looked at the code path and can't see any reason that registrations would be omitted when reindexing, which would be one reason the search might fail.

What version of FusionAuth are you running?

We're currently on 1.34.0.

What version of ES are you running?

That would be 6.8.16.

After reindex, if you search for a user with a registration (by something other than that nested search), is the registration data present?

No, the query wouldn't yield any result, it's like the user/document doesn't exist in the search index.

Any other searches fail?

As mentioned above, also searching directly via id for example fails.

Any other troubleshooting steps you can share?

Unfortunately not. Apologies, it's a tricky one!

Reindexing typically doesn't need to be done very often. Roughly how often are you reindexing (and why)?

I'd say we reindex once a month. Usually because we run into trouble with our user data. For example we migrate something in the user data and then data types have a mismatch and the ElasticSearch index is refusing to index the document. Then FusionAuth returns the "A request to the search index has failed. This error is unexpected. Contact Support." 503 and we need to "manually" fix this correcting the mistake in the user data and then running the reindex. It's all a bit suboptimal.

Thanks!

Any ideas?

yves

We use the user data in FusionAuth to store some additional attributes for our users.

The schema naturally evolves and so from time to time we need to migrate the schema, add attributes, remove attributes, convert items.

What are best practices on how to do this in FusionAuth?

Is there anything in FusionAuth supporting this? Or is the advice to do this outside of FusionAuth?

yves