FusionAuth
    • Home
    • Categories
    • Recent
    • Popular
    • Pricing
    • Contact us
    • Docs
    • Login
    1. Home
    2. tim.clark
    3. Posts
    T
    • Profile
    • Following 0
    • Followers 1
    • Topics 1
    • Posts 2
    • Best 0
    • Controversial 0
    • Groups 0

    Posts made by tim.clark

    • RE: "A request to the search index has failed" - OpenSearch(AWS)

      @mark-robustelli Hi Mark,

      Apologies for the delayed response. We've done some detailed investigation and can share concrete data.

      Environment:

      • FusionAuth 1.68.0
      • search.type=elasticsearch (per your docs, the correct setting for OpenSearch)
      • AWS-managed OpenSearch 2.19 (single t3.medium.search node)
      • This issue has persisted from at least version 1.55.1 (Dec 2025) through to current 1.68.0

      What we observe:

      Our deletion job calls the bulk delete API once per user:

      DELETE /api/user/bulk?queryString=<email>&hardDelete=true&dryRun=false

      On a typical night, 70–210 of these calls are made in rapid succession. A subset fail with:

      ERROR io.fusionauth.api.service.search.BaseElasticsearchSearchEngine - Failed to set a Point in Time on index [fusionauth_user]

      The response our application receives:

      {
      "generalErrors": [{"code": "[SearchEngineRequestFailedException]", "message": "A request to the search index has failed. This error is unexpected. Please review the troubleshooting guide..."}],
      "status": 503
      }

      CloudWatch metrics from our OpenSearch domain at the exact moment of failure (29 Jul, 23:12 UTC):

      │ Metric                        │ Value                                           │
      │ JVM memory pressure           │ 41%                                             │
      │ CPU utilisation               │ 11%                                             │
      │ Thread pool search rejections │ 0                                               │
      │ Thread pool write rejections  │ 0                                               │
      │ OpenSearch 5xx responses      │ 0                                               │
      │ OpenSearch 4xx responses      │ elevated (spike correlates with deletion burst) │
      │ HasUsedPointInTime            │ 0 (PIT never successfully created)              │
      │ Request volume                │ ~1,130 in 60 seconds vs normal ~15/min     |    
      

      The OpenSearch node is healthy and not under resource pressure. It appears to be actively rejecting the PIT creation request as a client error (4xx), not failing due to overload. The HasUsedPointInTime CloudWatch metric remains 0 throughout, confirming no PIT is ever successfully established.

      We don't have visibility into the raw 4xx response body from OpenSearch — FusionAuth wraps it in the generic SearchEngineRequestFailedException. Our OpenSearch application logs only capture JVM-level warnings, and audit logging isn't enabled.

      Questions:

      1. What PIT API endpoint does FusionAuth call internally when processing a queryString-based bulk delete? (e.g. POST /{index}/_pit vs POST /{index}/_search/point_in_time)
      2. Is it possible to expose the underlying search engine error response in the SearchEngineRequestFailedException? That would help diagnose whether this is a request format issue, a permissions issue,
        or something else specific to AWS-managed OpenSearch.
      3. Does FusionAuth use PIT when deleting by explicit userId array rather than queryString? If not, that might give us a workaround?

      Thanks,
      Tim

      posted in Q&A
      T
      tim.clark
    • "A request to the search index has failed" - OpenSearch(AWS)

      Hello,

      We're experiencing a known compatibility issue between FusionAuth and AWS-managed OpenSearch related to Point-in-Time (PIT) operations during bulk delete operations.

      Issue Summary:

      Error: SearchEngineRequestFailedException with message "A request to the search index has failed"
      
      Log Pattern: Unable to create point in time on index [fusionauth_user]
      
      Frequency: Occurs during nightly dedupe operations
      
      Impact: Prevents deletion of users via the bulk delete API
      
      Status Code: 503
      

      Environment:

      FusionAuth Version: 1.55.1
      
      Search Engine: AWS-managed OpenSearch
      
      Infrastructure: AWS
      

      Background:
      We understand this is a known issue with PIT operations against AWS OpenSearch, as documented in FusionAuth community discussions. We've reviewed release notes through version 1.62.1 and haven't found a specific fix for this issue.

      Questions:

      • Is there a timeline for a fix to address AWS OpenSearch PIT compatibility?
      • Are there alternative deletion methods that bypass PIT operations we could use as a workaround?

      We'd appreciate any guidance on addressing this issue or information about planned fixes.

      Thank you for your assistance.

      posted in Q&A
      T
      tim.clark