Searching Users With Elasticsearch

Overview

You can search users in FusionAuth to see how many there are, what they are doing, and more. This document will walk you through how to use FusionAuth’s powerful search capabilities to retrieve such user data.

Most of this document applies to searching Entities, which also leverage the Elasticsearch engine.

Examples are in curl, but you may also use any of the supported client libraries to run user searches in your favorite development language.

• What Type of Engine

• An Introduction To The Elasticsearch Search Engine

  • Three Types of Searches

  • Field Mappings

• Results Of a User Search

• User Search Examples

• Searching By User Ids

• Searching With queryString

• Searching With query

  • Searching On One Field

  • Searching On Multiple Fields

• Pagination

• Re-indexing

• Supported Elasticsearch Versions

• Elasticsearch And OpenSearch

• Troubleshooting Elasticsearch Queries

• Limitations

  • Maximum Users Returned Workarounds

  • Changing Data Field Types

• Additional Resources

What Type of Engine

This document discusses the Elasticsearch search engine.

You can determine which search engine FusionAuth is using. It will either be the database search engine or the elasticsearch search engine.

To do so, in the administrative user interface, navigate to System → About and scroll to the System section and you will see the configured search engine, as well as the version of Elasticsearch, if available. Below, it is Elasticsearch 7.17.0.

If you are using the database search engine, you’ll want to consult the database search engine documentation for more information.

• Learn more about each type of search engine.

• Learn more about switching search engines.

An Introduction To The Elasticsearch Search Engine

With any search, you can make either a POST or a GET request. The functionality is exactly the same, but a POST request can be larger. On the other hand, a GET request is easily shared. Pick what works for you. However, this document uses POST requests.

Results Of a User Search

The results of any search have the same format. There is a total object with the number of results and a users array containing user objects.

{
  "total": 1,
  "users": [
    {
      "active": true,
      "birthDate": "1976-05-30",
      "breachedPasswordLastCheckedInstant": 1471786483322,
      "data": {
        "displayName": "Johnny Boy",
        "favoriteColors": [
          "Red",
          "Blue"
        ]
      },
      "email": "example@fusionauth.io",
      "expiry": 1571786483322,
      "firstName": "John",
      "fullName": "John Doe",
      "id": "00000000-0000-0001-0000-000000000000",
      "imageUrl": "http://65.media.tumblr.com/tumblr_l7dbl0MHbU1qz50x3o1_500.png",
      "lastLoginInstant": 1471786483322,
      "lastName": "Doe",
      "memberships": [
        {
          "data": {
            "externalId": "cc6714c6-286c-411c-a6bc-ee413cda1dbc"
          },
          "groupId": "2cb5c83f-53ff-4d16-88bd-c5e3802111a5",
          "id": "27218714-305e-4408-bac0-23e7e1ddceb6",
          "insertInstant": 1471786482322
        }
      ],
      "middleName": "William",
      "mobilePhone": "303-555-1234",
      "passwordChangeRequired": false,
      "passwordLastUpdateInstant": 1471786483322,
      "preferredLanguages": [
        "en",
        "fr"
      ],
      "registrations": [
        {
          "applicationId": "10000000-0000-0002-0000-000000000001",
          "data": {
            "displayName": "Johnny",
            "favoriteSports": [
              "Football",
              "Basketball"
            ]
          },
          "id": "00000000-0000-0002-0000-000000000000",
          "insertInstant": 1446064706250,
          "lastLoginInstant": 1456064601291,
          "preferredLanguages": [
            "en",
            "fr"
          ],
          "roles": [
            "user",
            "community_helper"
          ],
          "username": "johnny123",
          "usernameStatus": "ACTIVE",
          "verified": true
        }
      ],
      "timezone": "America/Denver",
      "tenantId": "f24aca2b-ce4a-4dad-951a-c9d690e71415",
      "twoFactor": {
        "methods": [
          {
            "authenticator": {
              "algorithm": "HmacSHA1",
              "codeLength": 6,
              "timeStep": 30
            },
            "id": "35VW",
            "method": "authenticator"
          },
          {
            "id": "V7SH",
            "method": "sms",
            "mobilePhone": "555-555-5555"
          },
          {
            "email": "example@fusionauth.io",
            "id": "7K2G",
            "method": "email"
          }
        ]
      },
      "usernameStatus": "ACTIVE",
      "username": "johnny123",
      "verified": true
    }
  ]
}

User Search Examples

Below are examples of searches you can run. All examples are available to be run locally by downloading this GitHub repository and following the instructions in the README.

Each example uses the below shell script to run the search. Feel free to download it, update your API key and base FusionAuth URL, and experiment. The script uses the POST method so that the request body can be large. The value of the filename passed to the script changes each time, but everything else is the same.

#!/bin/sh

FILE=$1

API_KEY=90d8fb62-6f13-47d4-8ef6-1c3e687883c6
BASE_URL='http://localhost:9011'
#BASE_URL='https://sandbox.fusionauth.io'

curl -XPOST \
     -H 'Content-type: application/json' \
     -H "Authorization: $API_KEY" \
    $BASE_URL'/api/user/search' \
    -d @$FILE

Searching By User Ids

Here’s an example of an ids query.

{
  "search": {
    "ids": [
        "00000000-0000-0000-0000-000000000001",
        "00000000-0000-0000-0000-000000000002"
    ]
  }
}

This file is the first argument for the search.sh shell script. Here’s how you’d run the query if the above JSON is in the ids-request.json file.

./search.sh ids-request.json

Searching With queryString

Let’s look at some examples of queryString searches. Below, you are searching for dinesh, across all fields, and wildcards are prepended and appended.

{
  "search": {
    "numberOfResults": 50,
    "queryString": "dinesh",
    "sortFields": [
      {
        "missing": "_first",
        "name": "email",
        "order": "asc"
      }
    ],
    "startRow": 0
  }
}

Here’s how you’d run it if the above JSON is in all-fields-data-request.json.

./search.sh all-fields-data-request.json

Below, only the email field is searched for the string dinesh, because email: is specified.

{
  "search": {
    "numberOfResults": 50,
    "queryString": "email:dinesh*",
    "sortFields": [
      {
        "missing": "_first",
        "name": "email",
        "order": "asc"
      }
    ],
    "startRow": 0
  }
}

If you change the value from email:dinesh* to email:dinesh this query will return 0 results, because there is no user with that email and no wildcarding is done. If, instead, you change the value to email:dinesh@fusionauth.io this query will return the user.

Finally, below, the email field matches the string dinesh and the verified field is false. These are joined together with an OR clause which returns the user if either clause is true, but you can also use AND for the intersection.

{
  "search": {
    "numberOfResults": 50,
    "queryString": "email:dinesh* OR verified:true",
    "sortFields": [
      {
        "missing": "_first",
        "name": "email",
        "order": "asc"
      }
    ],
    "startRow": 0
  }
}

Learn more about making queryString queries in the Elasticsearch documentation.

Searching With query

The query parameter is the most powerful way to search. The query parameter is an escaped Elasticsearch query which is passed through to the corresponding Elasticsearch server. Let’s walk through some examples.

{
  "match": {
    "data.Company": {
      "query": "PiedPiper"
    }
  }
}

cat queryfile | jq -R -s '.'

"{\n  \"match\": {\n    \"data.Company\": {\n      \"query\": \"PiedPiper\"\n    }\n  }\n}\n"

{
  "search": {
    "numberOfResults": 50,
    "query":"{\n  \"match\": {\n    \"data.Company\": {\n      \"query\": \"PiedPiper\"\n    }\n  }\n}\n",
    "sortFields": [
      {
        "missing": "_first",
        "name": "email",
        "order": "asc"
      }
    ],
    "startRow": 0
  }
}

./search.sh user-data-simple-request.json

{
  "bool": {
    "must": [
      {
        "range": {
          "data.Salary": {
            "lt": 100000
          }
        }
      },
      {
        "match": {
          "data.Company": {
            "query": "PiedPiper"
          }
        }
      },
      {
        "match": {
          "verified": {
            "query": true
          }
        }
      },
      {
        "nested": {
          "path": "registrations",
          "query": {
            "bool": {
              "must": [
                {
                  "match": {
                    "registrations.data.paid": false
                  }
                }
              ]
            }
          }
        }
      }
    ]
  }
}

cat queryfile | jq -R -s '.'

"{\n  \"bool\": {\n    \"must\": [\n      {\n        \"range\": {\n          \"data.Salary\": {\n            \"lt\": 100000\n          }\n        }\n      },\n      {\n        \"match\": {\n          \"data.Company\": {\n            \"query\": \"PiedPiper\"\n          }\n        }\n      },\n      {\n        \"match\": {\n          \"verified\": {\n            \"query\": true\n          }\n        }\n      },\n      { \n        \"nested\" : {\n        \"path\" : \"registrations\",\n        \"query\" : {\n          \"bool\" : {\n            \"must\" : [ {\n              \"match\" : {\n                \"registrations.applicationId\" : \"e9fdb985-9173-4e01-9d73-ac2d60d1dc8e\"\n              },\n              \"match\" : {\n                \"registrations.data.paid\" : false\n              }\n            } ]\n          }\n        }\n      }\n      }\n    ]\n  }\n}\n"

{
  "search": {
    "numberOfResults": 50,
    "query": "{\n  \"bool\": {\n    \"must\": [\n      {\n        \"range\": {\n          \"data.Salary\": {\n            \"lt\": 100000\n          }\n        }\n      },\n      {\n        \"match\": {\n          \"data.Company\": {\n            \"query\": \"PiedPiper\"\n          }\n        }\n      },\n      {\n        \"match\": {\n          \"verified\": {\n            \"query\": true\n          }\n        }\n      },\n      { \n        \"nested\" : {\n        \"path\" : \"registrations\",\n        \"query\" : {\n          \"bool\" : {\n            \"must\" : [ {\n              \"match\" : {\n                \"registrations.applicationId\" : \"e9fdb985-9173-4e01-9d73-ac2d60d1dc8e\"\n              },\n              \"match\" : {\n                \"registrations.data.paid\" : false\n              }\n            } ]\n          }\n        }\n      }\n      }\n    ]\n  }\n}\n",
 "sortFields": [
      {
        "missing": "_first",
        "name": "email",
        "order": "asc"
      }
    ],
    "startRow": 0
  }
}

./search.sh user-data-complex-request.json

Pagination

You often need to paginate the results when running a query that matches many users or entities.

You can use the numberOfResults and startRow parameters to do so.

// this is pseudo code and won't work out of the box
// adapt for whichever client and programming language you are using

startRow = 0
numberOfResults = 25
fullresults = [] // new array
results = client.search(query, startRow, numberOfResults)
count = results.length
fullresults.append(results)

while (count > 0) {
  startRow = startRow + count
  results = client.search(query, startRow, numberOfResults)
  fullresults.append(results)
  count = results.length
}

// process fullresults here

You may also set numberOfResults to a higher number (500 or 5000, for example) to retrieve more results. However, processing results 25 or 50 at a time has less impact on the FusionAuth system.

Note that you’ll only be able to get back 10000 results no matter how you paginate. See the Limitations section for workarounds.

Re-indexing

It is possible, though rare, for an Elasticsearch index to become out of sync with the database. If you stand up FusionAuth with a database dump and restore, you may need to run this operation. You may also be instructed to do so by FusionAuth support.

In general, even if a temporary outage occurs with Elasticsearch, the index will be sync up automatically.

If you do need to run this, navigate to System → Reindex in the FusionAuth admin UI to initiate a reindex of all users. This navigation item will only be displayed when the search engine is Elasticsearch.

Optionally, you can also reindex via API.

• Rebuild Index API

• Index Status API

Supported Elasticsearch Versions

Versions 6.3.x - 7.17.x are currently supported.

Later versions may work as well but may not have been tested for compatibility.

OpenSearch version 2.x should also function properly with FusionAuth version >= 1.42.0.

Elasticsearch And OpenSearch

Currently, only Elasticsearch is officially supported.

Users and community members have reported some success with OpenSearch, the fork of Elasticsearch, but it is not currently supported.

Please see this GitHub issue for more details.

Troubleshooting Elasticsearch Queries

When you are troubleshooting Elasticsearch queries, running them against the Elasticsearch server is helpful.

When running FusionAuth locally in Docker, you can expose the Elasticsearch port by adding the following lines to your docker-compose.yml file and restarting.

# ... other stuff
  search:
# ... other stuff
    ports:
      - 9200:9200

Once you’ve opened up the port, you can query Elasticsearch directly. Here’s an example of a single field query.

curl -X GET "localhost:9200/_search?pretty" -H 'Content-Type: application/json' -d'
{
  "query": {
    "match": {
      "data.Company": {
        "query": "PiedPiper"
      }
    }
  }
}'

When you run this curl command, you’ll get back results similar to below.

{
  "took" : 6,
  "timed_out" : false,
  "_shards" : {
    "total" : 10,
    "successful" : 10,
    "skipped" : 0,
    "failed" : 0
  },
  "hits" : {
    "total" : {
      "value" : 3,
      "relation" : "eq"
    },
    "max_score" : 2.0,
    "hits" : [
      {
        "_index" : "fusionauth_user",
        "_type" : "_doc",
        "_id" : "00000000-0000-0000-0000-000000000003",
        "_score" : 2.0
      },
      {
        "_index" : "fusionauth_user",
        "_type" : "_doc",
        "_id" : "00000000-0000-0000-0000-000000000006",
        "_score" : 2.0
      },
      {
        "_index" : "fusionauth_user",
        "_type" : "_doc",
        "_id" : "00000000-0000-0000-0000-000000000005",
        "_score" : 2.0
      }
    ]
  }
}

When debugging, examine the hits.hits array. The _id value corresponds to the user Id that matched your query. If the content or number of users returned are different than expected, modify the query.

Limitations

You cannot filter search results in FusionAuth to only return certain fields. Instead you must do this through post-processing. So if you want to retrieve only the firstName and birthDate fields of a set of users, the results will give you each entire user object and you must select desired fields. You can use the JSON processing facilities in your chosen language to do so, or use a tool such as jq.

When using the Elasticsearch search engine, the maximum number of users returned for any search is 10000 users. There is an open bug tracking this issue.

Additional Resources

The FusionAuth Elasticsearch API documentation has examples of JSON for various queries, as well as additional supported parameters.

All search examples shown above can be run locally by downloading this GitHub repository and following the instructions in the README.