Topics tagged with elasticsearch

Elastic documentation is pretty abismal. Has anyone found better packages or workarounds?

dan — Invalid Date

@admin-9 I'm sorry to hear you're frustrated.

However, I don't know what you mean by a fleet server.

You have a couple of options:

you can test FusionAuth with the database search option, which doesn't require elasticsearch, but offers more limited search capabilities: https://fusionauth.io/docs/lifecycle/manage-users/search/search#using-the-database-search-engine you could use our docker compose examples which are preconfigured to work with opensearch or elasticsearch: https://fusionauth.io/docs/get-started/download-and-install/docker

If you can provide a few more details about what you are trying to accomplish, I might be able to offer more guidance.

After system reindex patching a user still returns SearchEngineRequestFailedException

croverw.norene8 — Invalid Date

If a user is still receiving a SearchEngineRequestFailedException after a system reindex and patching, there could be several reasons for this.

Firstly, the issue may not have been related to the index or patching, but rather a problem with the user's search query or the way they are interacting with the system.

Alternatively, it could be that the reindex and patching process did not fully resolve the underlying issue, and further investigation or troubleshooting is necessary.

It's also possible that there are other technical or environmental factors at play that are causing the exception to continue occurring. In any case, it may be necessary to consult with technical support or seek out additional resources to diagnose and resolve the issue.

See user and entity elasticsearch indexes when running in FusionAuth cloud

dan — Invalid Date

Elasticsearch access is not available for FusionAuth cloud deployments.

I would recommend running FusionAuth locally, which should display similar results as your cloud deployment for how the user and the entities are mapped.

Searching for user by email address returns a lot more results than I'd expect

dan — Invalid Date

Hiya.

You can see the elasticsearch query if you expand advanced in the UI.

Because of the way that we tokenize the search string, it is likely that a query like user@example.com will match more than just the user with the email address you are entering.

If you want to match only the email address in the UI, the easiest way to do it is to preface the query with email:. So email:user@example.com.

Hope that helps.

How can I see which fields are indexed?

dan — Invalid Date

We currently don't document these. Probably should; I'll put it on the list.

Until then, you can find the indexed fields by querying elasticsearch:

curl -XGET http://elasticsearchhost:port/fusionauth_user/_mapping

This will return something like the below JSON. This will vary based on your FusionAuth installation, however:

"fusionauth_user": { "mappings": { "_source": { "enabled": false }, "properties": { "active": { "type": "boolean" }, "birthDate": { "type": "date" }, "breachedPasswordLastCheckedInstant": { "type": "long" }, "breachedPasswordStatus": { "type": "text", "fields": { "keyword": { "type": "keyword", "ignore_above": 256 } } }, "data": { }, "email": { "type": "text", "analyzer": "exact_lower", "fielddata": true }, "fullName": { "type": "text", "fielddata": true }, "id": { "type": "keyword" }, "insertInstant": { "type": "date" }, "lastLoginInstant": { "type": "date" }, "lastUpdateInstant": { "type": "date" }, "login": { "type": "keyword" }, "memberships": { "type": "nested", "include_in_parent": true, "properties": { "data": { "type": "object" }, "groupId": { "type": "keyword" }, "id": { "type": "keyword" }, "insertInstant": { "type": "date" }, "userId": { "type": "keyword" } } }, "mobilePhone": { "type": "text", "fields": { "keyword": { "type": "keyword", "ignore_above": 256 } } }, "preferredLanguages": { "type": "text", "fields": { "keyword": { "type": "keyword", "ignore_above": 256 } } }, "registrations": { "type": "nested", "include_in_parent": true, "properties": { "applicationId": { "type": "keyword" }, "authenticationToken": { "type": "text", "fields": { "keyword": { "type": "keyword", "ignore_above": 256 } } }, "data": { }, "id": { "type": "keyword" }, "insertInstant": { "type": "date" }, "lastLoginInstant": { "type": "date" }, "lastUpdateInstant": { "type": "date" }, "preferredLanguages": { "type": "text", "fields": { "keyword": { "type": "keyword", "ignore_above": 256 } } }, "roles": { "type": "keyword" }, "tokens": { "properties": { "5ccff761-f809-49c7-a58c-27a3cb9ab650": { "type": "text", "fields": { "keyword": { "type": "keyword", "ignore_above": 256 } } }, "84ef0274-70db-44da-8762-aa6bcfbd2981": { "type": "text", "fields": { "keyword": { "type": "keyword", "ignore_above": 256 } } }, "Google": { "type": "text", "fields": { "keyword": { "type": "keyword", "ignore_above": 256 } } } } }, "username": { "type": "text", "fields": { "keyword": { "type": "keyword", "ignore_above": 256 } } }, "usernameStatus": { "type": "text", "fields": { "keyword": { "type": "keyword", "ignore_above": 256 } } }, "verified": { "type": "boolean" } } }, "tenantId": { "type": "keyword" }, "timezone": { "type": "text", "fields": { "keyword": { "type": "keyword", "ignore_above": 256 } } }, "username": { "type": "text", "fielddata": true }, "verified": { "type": "boolean" } } } } }

How do I get more results than the default allowed by ElasticSearch?

joshua — Invalid Date

You have a few options.

Limit Query One option is to find a way to limit your search by logical increments. For example, if you are searching on users you may consider obtaining and appending results by first name in a loop.

As an example:

for (letter in [a...z]) { query + " AND user email starts with $letter" // do work } Use Version > 1.24.0 when pulling something like users

https://fusionauth.io/docs/v1/tech/apis/users/#request-parameters-12

Request Parameters accurateTotal [Boolean] OPTIONAL Defaults to false AVAILABLE SINCE 1.24.0 Set this value equal to true to receive an accurate hit count on the API response. By default the search engine will limit the hit count to 10,000 users. This means that even if your query may match more than 10,000 users, the returned total count will be 10,000. This is adequate for many use cases such as pagination and general purpose queries. If you are looking for an accurate user count that can exceed 10,000 matches, you will want to set this value equal to true.

Added the wrong datatype to a user's data field

dan — Invalid Date

This exception is caused by the fact that your objects in elastic now have two different schemas (one with the number, one with the string). You need to manually fix that; one way to proceed might be to access elastic directly and remove that one user.

Then you'll need to reindex; here's more on that: https://fusionauth.io/docs/v1/tech/core-concepts/users/#reindex

Can you use the database search engine with millions of users?

dan — Invalid Date

I don't know of any issues. You’re mainly just giving up the ability to perform rich queries. The db search is a few LIKE statements.

I don't think we've tested millions running db search - but the only issue I can think of for normal usage would be the User page in the UI since we’ll be paginating over lots of users.

This should work ok - but pagination tends to slow way down as you get into higher pages,.

One can always flip on Elastic and run an index, so not much harm in trying it.

"missing search index" error message on install

dan — Invalid Date

If this is a new install, this is expected. During startup, FusionAuth goes into maintenance mode and reports back on the state of the system.

If you are in silent mode, the index will be created for you, if you are not in silent mode, FusionAuth will have entered an interactive startup mode and it is sitting on a prompt for you to create the Elasticsearch index.

Expand fields available for elasticsearch search engine

dan — Invalid Date

Not really, at least not through FusionAuth interfaces.

If you have particular queries that aren’t working well, you can open an issue in GitHub or a support ticket: https://github.com/FusionAuth/fusionauth-issues/issues

Any way to modify the elasticsearch index?

dan — Invalid Date

When you re-index, we delete our named index and rebuild it.

You would probably be better off creating your own index.

How can I find all users except one?

dan — Invalid Date

You can use the admin ui to build the query string.

Definitely test it out with the search first, and then use the bulk delete api.

Here's a shell script that searches for all users except one:

curl -vvv -XGET -H "Authorization: $API_KEY" 'http://localhost:9011/api/user/search/?queryString=NOT%20email:test%40example.com'

Note that I had to escape the space and the @ sign, but here's the elasticsearch query without the escaping: NOT email:test@example.com.

Also, if you are using the database search engine, the syntax will be entirely different, as this example relies on the Elasticsearch syntax.

Mastering in Elasticsearch

dan — Invalid Date

I'd start at https://www.elastic.co/guide/en/elasticsearch/reference/current/index.html and read the reference documentation.

HTH.

How can I protect my elasticsearch instances?

dan — Invalid Date

There are a few ways to do this.

This assumes that you are running elasticsearch on a different server than you are running the fusionauth instances. If they are on the same server, you should be fine, as that is the default configuration.

The first is at the network level, using a firewall or something like security groups on AWS. If you are doing this, you can configure the server that elasticsearch is installed on to accept requests only from the server that FusionAuth is installed on.

The second is to use basic authentication. That is, set fusionauth-search.servers in the fusionauth.properties file, or the FUSIONAUTH_SEARCH_SERVERS environment variable to include the basic username and password. https://user:password@example.com. And make sure to set up elastic to use basic auth, using whatever authentication source you'd like. (You could even go meta and have elasticsearch auth the user against the fusionauth instance 🙂 ).

Further discussion here.

I'm having an issue with Elastic Search queries in FusionAuth.

dan — Invalid Date

It depends on the issue, but there are two things you can do. Note that FusionAuth typically sends the query you post to the /api/user/search endpoint with the query parameter straight through to Elasticsearch

First, ensure you are running elasticsearch and not the database search engine.

Then, for some queries you can see the generate ElasticSearch query strings by clicking on the 'advanced' option in the user search area of the admin UI. This can be helpful.

Finally, try running the query directly against elasticsearch and seeing if it works. Here are examples which will pull back all the data in your elasticsearch cluster (beware!).

curl -XPOST -H 'Content-type: application/json' "http://localhost:9021/_search" -d' { "query": { "match_all": {} } }'

or, if you have the query stored in query.json

curl -XPOST -H 'Content-type: application/json' "http://localhost:9021/_search" -d@query.json

multi_match query?

dan — Invalid Date

It looks like multi match (at least as of 2019) doesn't support wildcards:

[a] multimatch query requires match queries as its internal queries. And a match query doesn't support wildcards.

https://discuss.elastic.co/t/query-with-multimatch-and-wildcard/189962

Can you try with a non wildcard query?

"A request to the search index has failed. This error is unexpected."

dan — Invalid Date

Turns out that I didn't have much disk space.

Saw this in the FA search logs:

{"type": "server", "timestamp": "2020-06-10T14:34:28,378Z", "level": "WARN", "component": "o.e.c.r.a.DiskThresholdMonitor", "cluster.name": "FusionAuth", "node.name": "ubuntu-xenial", "message": "flood stage disk watermark [95%] exceeded on [NE_DhFssRru-H5oIiLwBjA][ubuntu-xenial][/usr/local/fusionauth/data/search/esv6/nodes/0] free: 887.9mb[4.4%], all indices on this node will be marked read-only", "cluster.uuid": "LGdaij30RrS4FZbgsT_6KA", "node.id": "NE_DhFssRru-H5oIiLwBjA" }

Deleting some things from my hard drive lowered my disk usage below 95% and I no longer saw the message.

Do you support sig4 auth headers for AWS Elasticsearch?

dan — Invalid Date

If you are using https://aws.amazon.com/elasticsearch-service/ for your Elasticsearch server, you can access it via AWS APIs and use IAM to control access: https://docs.aws.amazon.com/elasticsearch-service/latest/developerguide/es-ac.html

However, FusionAuth doesn't currently support the AWS signature for Elasticsearch requests.

The recommended way of securing such clusters is to place it in a private subnet and restricting traffic to it using a security group. More information: https://fusionauth.io/docs/v1/tech/installation-guide/securing#fusionauth-search

If you have to make it public to make it accessible to resources outside if AWS you could use a source IP lock, a VPN, basic auth if AWS supports it, or you could proxy the request perhaps to another endpoint that can build the AWS sig v4 header.