ElasticSearch reindex doesn't reindex all documents

yves

I've got a question regarding the reindexing of the ElasticSearch fusionauth_user index.

When we perform a reindex it seems that some users/documents get "lost".

Some observations:

We've enabled debugging on ElasticSearch and don't see any errors being thrown when the index is dropped and recreated.
The query that fails afterwards is a "nested" search for a particular registration users have.
When we PATCH or lock/unlock users then they appear in the ElasticSearch index again and the query succeeds.

What happens when the reindex is performed? I'm struggling to understand why or how this could happen because I assumed all users/documents are inserted in the new index again?

Right now our workaround is to PATCH all users so that we can be sure that they're in the index.

Any ideas? Pointers?

dan

Hmmm. I looked at the code path and can't see any reason that registrations would be omitted when reindexing, which would be one reason the search might fail.

What version of FusionAuth are you running?
What version of ES are you running?
After reindex, if you search for a user with a registration (by something other than that nested search), is the registration data present?
Any other searches fail?
Any other troubleshooting steps you can share?
Reindexing typically doesn't need to be done very often. Roughly how often are you reindexing (and why)?

Thanks!

yves

Thanks for the reply @dan!

Will answer inline.

@dan said in ElasticSearch reindex doesn't reindex all documents:

Hmmm. I looked at the code path and can't see any reason that registrations would be omitted when reindexing, which would be one reason the search might fail.

What version of FusionAuth are you running?

We're currently on 1.34.0.

What version of ES are you running?

That would be 6.8.16.

After reindex, if you search for a user with a registration (by something other than that nested search), is the registration data present?

No, the query wouldn't yield any result, it's like the user/document doesn't exist in the search index.

Any other searches fail?

As mentioned above, also searching directly via id for example fails.

Any other troubleshooting steps you can share?

Unfortunately not. Apologies, it's a tricky one!

Reindexing typically doesn't need to be done very often. Roughly how often are you reindexing (and why)?

I'd say we reindex once a month. Usually because we run into trouble with our user data. For example we migrate something in the user data and then data types have a mismatch and the ElasticSearch index is refusing to index the document. Then FusionAuth returns the "A request to the search index has failed. This error is unexpected. Contact Support." 503 and we need to "manually" fix this correcting the mistake in the user data and then running the reindex. It's all a bit suboptimal.

Thanks!

Any ideas?

dan

@yves

I'd be interested to see if the same issue occurs on the latest version of FusionAuth. There's 4-5 mentions of Elasticsearch in the release notes between 1.34.0 and 1.45.3, but none of them appear to address what you are seeing.

As mentioned above, also searching directly via id for example fails.

Sorry, I missed that. Are you saying when you search using the ids method (described here under the heading Search for Users by Id) it doesn't find these users? Because that search should go directly to the database and not through ES at all.

It's all a bit suboptimal.

Off topic, but please consider adding your voice to this issue which might help with your schema changes.