Prometheus is dropping samples with duplicate timestamps

dalamenona

Hello every one,
it seems that once in a while FusionAuth is returning duplicate metrics but reporting different values. Our Prometheus stack is deployed on k8s and we get this error:

{"caller":"scrape.go:1820","component":"scrape manager","level":"warn","msg":"Error on ingesting samples with different value but same timestamp","num_dropped":8,"scrape_pool":"serviceMonitor/dev/fusionauth-dev/0","target":"http://xxx.xxx.xxx.244:9011/api/prometheus/metrics","ts":"2025-12-08T09:34:49.026Z"}
{"caller":"scrape.go:1820","component":"scrape manager","level":"warn","msg":"Error on ingesting samples with different value but same timestamp","num_dropped":8,"scrape_pool":"serviceMonitor/dev/fusionauth-dev/0","target":"http://xxx.xxx.xxx.244:9011/api/prometheus/metrics","ts":"2025-12-08T09:35:19.020Z"}
{"caller":"scrape.go:1820","component":"scrape manager","level":"warn","msg":"Error on ingesting samples with different value but same timestamp","num_dropped":8,"scrape_pool":"serviceMonitor/dev/fusionauth-dev/0","target":"http://xxx.xxx.xxx.244:9011/api/prometheus/metrics","ts":"2025-12-08T09:35:49.029Z"}

This error happens randomly and it doesn't tell which metrics are duplicates.
Is anyone facing the issue in subject while scraping FusionAuth metrics with Prometheus?
Thank you for any hint
have a lovely day
_
Fabio

mark.robustelli

@dalamenona This error is coming from Prometheus right? Is there a way to get it to tell you which metric is being reported? If not, could you set up a network monitor and capture the traffic that is being sent to narrow down the metric being sent by FusionAuth that is causing the problem? Maybe then we can look into why FusionAuth is sending the conflicting data.