I am trying to setup K6 output to a multi-tenant Mimir deployment via “experimental-prometheus-rw”
I have deployed multi-tenancy enabled Mimir OpenSource in K8s and grafana-agents running on different clusters can send metrics using X-Scope-OrgID header to different tenants successfully.
I cannot manage to set it up for K6 Prometheus Remote Write with “-o experimental-prometheus-rw” option. I have set following variables:
and I see the following error at the end of K6 run:
ERRO Failed to send the time series data to the endpoint error=“got status code: 401 instead expected a 2xx successful status code” output=“Prometheus remote write”
If I unset “K6_PROMETHEUS_RW_HEADERS_X-Scope-OrgID” environment variable, I see the metrics stored as “anonymous” tenant (default tenant on Mimir if no X-Scope-OrgID defined)
Anyone used Prometheus Remote Write with a multi-tenant Mimir deployment to a specific tenant?
Thanks in advance.
welcome to the community forum
It seems you’re getting an unexpected error because at the moment the scope id header is not supported as you can check from this issue. Better API for HTTP Headers · Issue #101 · grafana/xk6-output-prometheus-remote · GitHub
Please, add a thumbs up reaction to the issue so we can raise the priority.
Are you sure you’re setting it? You should get an error from k6 similar to this:
level=error msg="invalid environment variable name 'K6_PROMETHEUS_RW_HEADERS_X-Scope-OrgId'"
Thanks for the link. There is a slight difference though with the issue:
Documentation says env variable should be K6_PROMETHEUS_RW_HEADERS_=
So I have an env variable as stated in my question:
But the issue has
So which one should I use? I can try that as well
Thanks in advance…
Only if I use
I got that error about status code 401 at the end.
Other combinations I have tested below did not give any error but did not do anything neither
I never get that invalid environment variable error.