Searching Users With Elasticsearch

Overview

You can search users in FusionAuth to see how many there are, what they are doing, and more. This document will walk you through how to use FusionAuth’s powerful search capabilities to retrieve such user data.

Most of this document applies to searching Entities, which also leverage the Elasticsearch engine.

Examples are in curl, but you may also use any of the supported client libraries to run user searches in your favorite development language.

What Type of Engine
An Introduction To The Elasticsearch Search Engine
- Three Types of Searches
- Field Mappings
Results Of a User Search
User Search Examples
Searching By User Ids
Searching With queryString
Searching With query
- Searching On One Field
- Searching On Multiple Fields
Pagination
Re-indexing
Supported Elasticsearch Versions
Elasticsearch And OpenSearch
Troubleshooting Elasticsearch Queries
Limitations
- Maximum Users Returned Workarounds
Additional Resources

What Type of Engine

This document discusses the Elasticsearch search engine.

You can determine which search engine FusionAuth is using. It will either be the database search engine or the elasticsearch search engine.

To do so, in the administrative user interface, navigate to System → About and scroll to the System section and you will see the configured search engine, as well as the version of Elasticsearch, if available. Below, it is Elasticsearch 7.17.0.

If you are using the database search engine, you’ll want to consult the database search engine documentation for more information.

Learn more about each type of search engine.
Learn more about switching search engines.

An Introduction To The Elasticsearch Search Engine

With any search, you can make either a POST or a GET request. The functionality is exactly the same, but a POST request can be larger. On the other hand, a GET request is easily shared. Pick what works for you. However, this document uses POST requests.

Three Types of Searches

There are three parameters for a search, and they are mutually exclusive.

Table 1. Search Type Summary
Parameter	Uses Elasticsearch	Best For
`ids`	No	When you know the exact Ids of users you are trying to retrieve. You may have stored the id off in some other datastore, or be responding to a webhook.
`queryString`	Yes	When you are searching on one field or want to search across multiple fields for strings.
`query`	Yes	Useful to leverage the full power of Elasticsearch queries and need to look across different fields, including nested values, and/or use compound or complex search parameters such as ranges.

Ids Searches

ids searches are useful when you know exactly which users you want to retrieve. This is the only search which is guaranteed to only query the database, never Elasticsearch.

QueryString Searches

The queryString search is a case-insensitive match. Whitespace is allowed in the search, but must be URL escaped; for example, using %20 for a space character. Elasticsearch compatible regular expressions may be used, so you may search by prefix or suffix using the * wildcard. You may also search particular fields by prefixing the query with a field name, such as email:.

If the queryString parameter has a : it will be passed as-is. Otherwise, if it has no spaces and contains a @ it is assumed to be an email address and will be passed with a prefix: email:. Otherwise, if it has no spaces it will have the wildcard * prepended and appended.

You may use AND and OR clauses in this parameter to construct compound queries.

Query Searches

The query parameter search requires an escaped JSON string which is passed to Elasticsearch and therefore must be a valid Elasticsearch query. With a query search, you have the full power of the Elasticsearch query language.

Field Mappings

For both queryString and query searches, you may search against specific fields.

The exact list of indexed fields is not documented, but if you are running FusionAuth self-hosted, you can find the list by retrieving the Elasticsearch mapping.

If you are using FusionAuth Cloud, please open a support ticket if you have a question about which fields are searchable.

Fields of the user object are indexed, as are the following relationships, which are available using the nested query type.

memberships - group memberships for this user
registrations - application registrations for this user

Learn more about the nested query type.

Results Of a User Search

The results of any search have the same format. There is a total object with the number of results and a users array containing user objects.

Sample Results From a Search


{
  "total": 1,
  "users": [
    {
      "active": true,
      "birthDate": "1976-05-30",
      "breachedPasswordLastCheckedInstant": 1471786483322,
      "data": {
        "displayName": "Johnny Boy",
        "favoriteColors": [
          "Red",
          "Blue"
        ]
      },
      "email": "example@fusionauth.io",
      "expiry": 1571786483322,
      "firstName": "John",
      "fullName": "John Doe",
      "id": "00000000-0000-0001-0000-000000000000",
      "imageUrl": "http://65.media.tumblr.com/tumblr_l7dbl0MHbU1qz50x3o1_500.png",
      "lastLoginInstant": 1471786483322,
      "lastName": "Doe",
      "memberships": [{
        "data": {
          "externalId": "cc6714c6-286c-411c-a6bc-ee413cda1dbc"
        },
        "groupId": "2cb5c83f-53ff-4d16-88bd-c5e3802111a5",
        "id": "27218714-305e-4408-bac0-23e7e1ddceb6",
        "insertInstant": 1471786482322
      }],
      "middleName": "William",
      "mobilePhone": "303-555-1234",
      "passwordChangeRequired": false,
      "passwordLastUpdateInstant": 1471786483322,
      "preferredLanguages": [
        "en",
        "fr"
      ],
      "registrations": [
        {
          "applicationId": "10000000-0000-0002-0000-000000000001",
          "data": {
            "displayName": "Johnny",
            "favoriteSports": [
              "Football",
              "Basketball"
            ]
          },
          "id": "00000000-0000-0002-0000-000000000000",
          "insertInstant": 1446064706250,
          "lastLoginInstant": 1456064601291,
          "preferredLanguages": [
            "en",
            "fr"
          ],
          "roles": [
            "user",
            "community_helper"
          ],
          "username": "johnny123",
          "usernameStatus": "ACTIVE",
          "verified": true
        }
      ],
      "timezone": "America/Denver",
      "tenantId": "f24aca2b-ce4a-4dad-951a-c9d690e71415",
      "twoFactor": {
        "methods": [
          {
            "authenticator": {
              "algorithm": "HmacSHA1",
              "codeLength": 6,
              "timeStep": 30
            },
            "id": "35VW",
            "method": "authenticator"
          },
          {
            "id": "V7SH",
            "method": "sms",
            "mobilePhone": "555-555-5555"
          },
          {
            "email": "example@fusionauth.io",
            "id": "7K2G",
            "method": "email"
          }
        ]
      },
      "usernameStatus": "ACTIVE",
      "username": "johnny123",
      "verified": true
    }
  ]
}

User Search Examples

Below are examples of searches you can run. All examples are available to be run locally by downloading this GitHub repository and following the instructions in the README.

Each example uses the below shell script to run the search. Feel free to download it, update your API key and base FusionAuth URL, and experiment. The script uses the POST method so that the request body can be large. The value of the filename passed to the script changes each time, but everything else is the same.

search.sh: An Example Shell Script For Searching


#!/bin/sh

FILE=$1

API_KEY=90d8fb62-6f13-47d4-8ef6-1c3e687883c6
BASE_URL='http://localhost:9011'
#BASE_URL='https://sandbox.fusionauth.io'

curl -XPOST \
     -H 'Content-type: application/json' \
     -H "Authorization: $API_KEY" \
    $BASE_URL'/api/user/search' \
    -d @$FILE

Searching By User Ids

Here’s an example of an ids query.

Example Request JSON Searching By ids


{
  "search": {
    "ids": [
        "00000000-0000-0000-0000-000000000001",
        "00000000-0000-0000-0000-000000000002"
    ]
  }
}

This file is the first argument for the search.sh shell script. Here’s how you’d run the query if the above JSON is in the ids-request.json file.

Running The Example Shell Script


./search.sh ids-request.json

Searching With `queryString`

Let’s look at some examples of queryString searches. Below, you are searching for dinesh, across all fields, and wildcards are prepended and appended.

Example Request JSON For Searching Across All Indexed Fields With queryString


{
  "search": {
    "numberOfResults": 50,
    "queryString": "dinesh",
    "sortFields": [
      {
        "missing": "_first",
        "name": "email",
        "order": "asc"
      }
    ],
    "startRow": 0
  }
}

Here’s how you’d run it if the above JSON is in all-fields-data-request.json.

Running The Example Shell Script


./search.sh all-fields-data-request.json

Below, only the email field is searched for the string dinesh, because email: is specified.

Example Request JSON For Searching By Emails With queryString


{
  "search": {
    "numberOfResults": 50,
    "queryString": "email:dinesh*",
    "sortFields": [
      {
        "missing": "_first",
        "name": "email",
        "order": "asc"
      }
    ],
    "startRow": 0
  }
}

If you change the value from email:dinesh* to email:dinesh this query will return 0 results, because there is no user with that email and no wildcarding is done. If, instead, you change the value to email:dinesh@fusionauth.io this query will return the user.

Finally, below, the email field matches the string dinesh and the verified field is false. These are joined together with an OR clause which returns the user if either clause is true, but you can also use AND for the intersection.

Example Request JSON For Searching By Emails And The Verified Field


{
  "search": {
    "numberOfResults": 50,
    "queryString": "email:dinesh* OR verified:true",
    "sortFields": [
      {
        "missing": "_first",
        "name": "email",
        "order": "asc"
      }
    ],
    "startRow": 0
  }
}

Learn more about making queryString queries in the Elasticsearch documentation.

Searching With `query`

The query parameter is the most powerful way to search. The query parameter is an escaped Elasticsearch query which is passed through to the corresponding Elasticsearch server. Let’s walk through some examples.

Searching On One Field

First, build the Elasticsearch query. Below is a query matching all users with a user.data.Company attribute of PiedPiper.

The FusionAuth user.data field can hold arbitrary JSON and is useful if your users have customer data fields.

There is a similar .data field on many FusionAuth objects, though not all of them are indexed and searchable.

The key is case sensitive, so searching on user.data.company will return zero results. However, the value is case insensitive. Searching for values of PiedPiper, piedpiper and PIEDPIPER will return the same number of results.

Example Elasticsearch Query JSON Searching the Company User Attribute


{
  "match": {
    "data.Company": {
      "query": "PiedPiper"
    }
  }
}

Next, escape any JSON characters in the string. This is a major difference when using the query method when compared to the queryString method. Because you can use the full Elasticsearch query language with the query method, but FusionAuth’s APIs also expect JSON, you must escape the Elasticsearch query JSON.

One option to escape the JSON is to use jq. If the Elasticsearch query above is in queryfile, escape it using the below command. (You may ignore or remove the newlines indicated by \n; they’ll be ignored by Elasticsearch.)

Escaping Elasticsearch Query String using jq


cat queryfile | jq -R -s '.'

That command outputs this string.

Example Escaped Elasticsearch Query JSON For Searching the Company User Attribute


"{\n  \"match\": {\n    \"data.Company\": {\n      \"query\": \"PiedPiper\"\n    }\n  }\n}\n"

Next, add that string as the query parameter to a request.

Example Request JSON For Searching By the Company User Attribute Emails With queryString


{
  "search": {
    "numberOfResults": 50,
    "query":"{\n  \"match\": {\n    \"data.Company\": {\n      \"query\": \"PiedPiper\"\n    }\n  }\n}\n",
    "sortFields": [
      {
        "missing": "_first",
        "name": "email",
        "order": "asc"
      }
    ],
    "startRow": 0
  }
}

Then, run the query using the search.sh script. Here’s how you’d run it if the above JSON is in user-data-simple-request.json.

Running The Example Shell Script


./search.sh user-data-simple-request.json

Searching On Multiple Fields

You can also perform advanced searches, including with multiple clauses, nested queries and numeric ranges. Let’s take a closer look at a more complex query.

First, here’s the query.

Example Elasticsearch Complex Query JSON


{
  "bool": {
    "must": [
      {
        "range": {
          "data.Salary": {
            "lt": 100000
          }
        }
      },
      {
        "match": {
          "data.Company": {
            "query": "PiedPiper"
          }
        }
      },
      {
        "match": {
          "verified": {
            "query": true
          }
        }
      },
      {
        "nested": {
          "path": "registrations",
          "query": {
            "bool": {
              "must": [
                {
                  "match": {
                    "registrations.data.paid": false
                  }
                }
              ]
            }
          }
        }
      }
    ]
  }
}

With this query, you are searching for users who meet the following criteria.

A salary of less than 100000.
Pied Piper employment.
A verified email address.
Registered for a particular application in your system. The registrations object is nested.
For that particular application, having a paid attribute of false.

This is quite specific. In addition to matching a value in the user.data field like the first example, you are searching ranges, checking standard user attributes such as verified, finding users who have registered for a given application, and examining application specific fields.

With the query search method, if you can write the Elasticsearch query, you can find your users.

After creating the query, escape JSON characters in the query string. You can use jq to do the escaping. If the Elasticsearch query above is in queryfile, you can escape it using this command.

Escaping Elasticsearch Query String using jq


cat queryfile | jq -R -s '.'

That will display this escaped string:

Example Escaped Elasticsearch Complex Query JSON


"{\n  \"bool\": {\n    \"must\": [\n      {\n        \"range\": {\n          \"data.Salary\": {\n            \"lt\": 100000\n          }\n        }\n      },\n      {\n        \"match\": {\n          \"data.Company\": {\n            \"query\": \"PiedPiper\"\n          }\n        }\n      },\n      {\n        \"match\": {\n          \"verified\": {\n            \"query\": true\n          }\n        }\n      },\n      { \n        \"nested\" : {\n        \"path\" : \"registrations\",\n        \"query\" : {\n          \"bool\" : {\n            \"must\" : [ {\n              \"match\" : {\n                \"registrations.applicationId\" : \"e9fdb985-9173-4e01-9d73-ac2d60d1dc8e\"\n              },\n              \"match\" : {\n                \"registrations.data.paid\" : false\n              }\n            } ]\n          }\n        }\n      }\n      }\n    ]\n  }\n}\n"

The next step is to add that string as the query parameter to a request.

Example Request JSON For a Complex Query


{
  "search": {
    "numberOfResults": 50,
    "query": "{\n  \"bool\": {\n    \"must\": [\n      {\n        \"range\": {\n          \"data.Salary\": {\n            \"lt\": 100000\n          }\n        }\n      },\n      {\n        \"match\": {\n          \"data.Company\": {\n            \"query\": \"PiedPiper\"\n          }\n        }\n      },\n      {\n        \"match\": {\n          \"verified\": {\n            \"query\": true\n          }\n        }\n      },\n      { \n        \"nested\" : {\n        \"path\" : \"registrations\",\n        \"query\" : {\n          \"bool\" : {\n            \"must\" : [ {\n              \"match\" : {\n                \"registrations.applicationId\" : \"e9fdb985-9173-4e01-9d73-ac2d60d1dc8e\"\n              },\n              \"match\" : {\n                \"registrations.data.paid\" : false\n              }\n            } ]\n          }\n        }\n      }\n      }\n    ]\n  }\n}\n",
 "sortFields": [
      {
        "missing": "_first",
        "name": "email",
        "order": "asc"
      }
    ],
    "startRow": 0
  }
}

Run the query using the search.sh script. Here’s how you’d run it if the above JSON is in user-data-complex-request.json.

Running The Example Shell Script


./search.sh user-data-complex-request.json

You’ll get back any users that match all the criteria.

Pagination

You often need to paginate the results when running a query that matches many users or entities.

You can use the numberOfResults and startRow parameters to do so.

Pagination pseudocode to retrieve search results



// this is pseudo code and won't work out of the box
// adapt for whichever client and programming language you are using

startRow = 0
numberOfResults = 25
fullresults = [] // new array
results = client.search(query, startRow, numberOfResults)
count = results.length
fullresults.append(results)

while (count > 0) {
  startRow = startRow + count
  results = client.search(query, startRow, numberOfResults)
  fullresults.append(results)
  count = results.length
}

// process fullresults here

You may also set numberOfResults to a higher number (500 or 5000, for example) to retrieve more results. However, processing results 25 or 50 at a time has less impact on the FusionAuth system.

Note that you’ll only be able to get back 10000 results no matter how you paginate. See the Limitations section for workarounds.

Re-indexing

Reindexing is an expensive operation, especially if your system has a large number of users, so it should not be run unless necessary.

It is possible, though rare, for an Elasticsearch index to become out of sync with the database. If you stand up FusionAuth with a database dump and restore, you may need to run this operation. You may also be instructed to do so by FusionAuth support.

In general, even if a temporary outage occurs with Elasticsearch, the index will be sync up automatically.

If you do need to run this, navigate to System → Reindex in the FusionAuth admin UI to initiate a reindex of all users. This navigation item will only be displayed when the search engine is Elasticsearch.

Optionally, you can also reindex via API.

Supported Elasticsearch Versions

Versions 6.3.x - 7.17.x are currently supported.

Later versions may work as well but may not have been tested for compatibility.

Elasticsearch And OpenSearch

Currently, only Elasticsearch is officially supported.

Users and community members have reported some success with OpenSearch, the fork of Elasticsearch, but it is not currently supported.

Please see this GitHub issue for more details.

Troubleshooting Elasticsearch Queries

When you are troubleshooting Elasticsearch queries, running them against the Elasticsearch server is helpful. If you are running in FusionAuth Cloud, however, you cannot access Elasticsearch.

When running FusionAuth locally in Docker, you can expose the Elasticsearch port by adding the following lines to your docker-compose.yml file and restarting.

Exposing Elasticsearch Ports


# ... other stuff
  search:
# ... other stuff
    ports:
      - 9200:9200

Once you’ve opened up the port, you can query Elasticsearch directly. Here’s an example of a single field query.

Example Of Querying Elasticsearch Directly


curl -X GET "localhost:9200/_search?pretty" -H 'Content-Type: application/json' -d'
{
  "query": {
    "match": {
      "data.Company": {
        "query": "PiedPiper"
      }
    }
  }
}'

When you run this curl command, you’ll get back results similar to below.

Results Of Querying Elasticsearch Directly


{
  "took" : 6,
  "timed_out" : false,
  "_shards" : {
    "total" : 10,
    "successful" : 10,
    "skipped" : 0,
    "failed" : 0
  },
  "hits" : {
    "total" : {
      "value" : 3,
      "relation" : "eq"
    },
    "max_score" : 2.0,
    "hits" : [
      {
        "_index" : "fusionauth_user",
        "_type" : "_doc",
        "_id" : "00000000-0000-0000-0000-000000000003",
        "_score" : 2.0
      },
      {
        "_index" : "fusionauth_user",
        "_type" : "_doc",
        "_id" : "00000000-0000-0000-0000-000000000006",
        "_score" : 2.0
      },
      {
        "_index" : "fusionauth_user",
        "_type" : "_doc",
        "_id" : "00000000-0000-0000-0000-000000000005",
        "_score" : 2.0
      }
    ]
  }
}

When debugging, examine the hits.hits array. The _id value corresponds to the user Id that matched your query. If the content or number of users returned are different than expected, modify the query.

Limitations

You cannot filter search results in FusionAuth to only return certain fields. Instead you must do this through post-processing. So if you want to retrieve only the firstName and birthDate fields of a set of users, the results will give you each entire user object and you must select desired fields. You can use the JSON processing facilities in your chosen language to do so, or use a tool such as jq.

When using the Elasticsearch search engine, the maximum number of users returned for any search is 10000 users. There is an open bug tracking this issue.

Maximum Users Returned Workarounds

You can work around the user search limit by writing one or more search queries to return less than 10000 users. You can know if the query returns more than that limit by using the accurateTotal request parameter. The User Search API contains more information about the accurateTotal parameter and its effect.

For example, if you needed to download all users, you could query for users with an email address starting with A, then starting with B, and so on. Here’s a sample shell script which retrieves all users using this strategy.

If you have access to the Elasticsearch server, you may also run your query directly against it. The Elasticsearch query result will return Elasticsearch document Ids. Each document Id corresponds to a FusionAuth user Id. To retrieve user data, which is most likely what you are after, use the User Search API with one or more user Ids.

Additional Resources

The FusionAuth Elasticsearch API documentation has examples of JSON for various queries, as well as additional supported parameters.

All search examples shown above can be run locally by downloading this GitHub repository and following the instructions in the README.

Searching Users With Elasticsearch

Overview

What Type of Engine

An Introduction To The Elasticsearch Search Engine

Three Types of Searches

Ids Searches

QueryString Searches

Query Searches

Field Mappings

Results Of a User Search

User Search Examples

Searching By User Ids

Searching With `queryString`

Searching With `query`

Searching On One Field

Searching On Multiple Fields

Re-indexing

Supported Elasticsearch Versions

Elasticsearch And OpenSearch

Troubleshooting Elasticsearch Queries

Limitations

Maximum Users Returned Workarounds

Additional Resources

Feedback

How helpful was this page?

See a problem?

Have a question or comment to share?

Searching Users With Elasticsearch

Overview

What Type of Engine

An Introduction To The Elasticsearch Search Engine

Three Types of Searches

Ids Searches

QueryString Searches

Query Searches

Field Mappings

Results Of a User Search

User Search Examples

Searching By User Ids

Searching With queryString

Searching With query

Searching On One Field

Searching On Multiple Fields

Pagination

Re-indexing

Supported Elasticsearch Versions

Elasticsearch And OpenSearch

Troubleshooting Elasticsearch Queries

Limitations

Maximum Users Returned Workarounds

Additional Resources

Feedback

How helpful was this page?

See a problem?

Have a question or comment to share?

Searching With `queryString`

Searching With `query`