Experimental-prometheus-rw output to a multi-tenant Mimir deployment

Grafana k6 Extensions
1

Hi all,

I am trying to setup K6 output to a multi-tenant Mimir deployment via “experimental-prometheus-rw”

I have deployed multi-tenancy enabled Mimir OpenSource in K8s and grafana-agents running on different clusters can send metrics using X-Scope-OrgID header to different tenants successfully.

I cannot manage to set it up for K6 Prometheus Remote Write with “-o experimental-prometheus-rw” option. I have set following variables:

K6_PROMETHEUS_RW_PASSWORD=<masked_password>
K6_PROMETHEUS_RW_SERVER_URL=https://<masked_fqdn>/api/v1/push
K6_PROMETHEUS_RW_USERNAME=
K6_PROMETHEUS_RW_HEADERS_X-Scope-OrgID=<tenant_id>

and I see the following error at the end of K6 run:

ERRO[0001] Failed to send the time series data to the endpoint error=“got status code: 401 instead expected a 2xx successful status code” output=“Prometheus remote write”

If I unset “K6_PROMETHEUS_RW_HEADERS_X-Scope-OrgID” environment variable, I see the metrics stored as “anonymous” tenant (default tenant on Mimir if no X-Scope-OrgID defined)

Anyone used Prometheus Remote Write with a multi-tenant Mimir deployment to a specific tenant?

Thanks in advance.
Suleyman Kutlu

4

Hi @snk,
welcome to the community forum :tada:

It seems you’re getting an unexpected error because at the moment the scope id header is not supported as you can check from this issue. Better API for HTTP Headers · Issue #101 · grafana/xk6-output-prometheus-remote · GitHub
Please, add a thumbs up reaction to the issue so we can raise the priority.

Are you sure you’re setting it? You should get an error from k6 similar to this:

level=error msg="invalid environment variable name 'K6_PROMETHEUS_RW_HEADERS_X-Scope-OrgId'"
1 Like
6

Hi @codebien

Thanks for the link. There is a slight difference though with the issue:

Documentation says env variable should be K6_PROMETHEUS_RW_HEADERS_=

So I have an env variable as stated in my question:

K6_PROMETHEUS_RW_HEADERS_X-Scope-OrgID=<tenant_id>

But the issue has

K6_PROMETHEUES_RW_HTTP_HEADERS=X-Scope-OrgID:<org>,X-ANOTHER_HEADER:val

So which one should I use? I can try that as well :wink:

Thanks in advance…
-Suleyman

7

Only if I use

K6_PROMETHEUS_RW_HEADERS_X-Scope-OrgID=<tenant_id>

I got that error about status code 401 at the end.

Other combinations I have tested below did not give any error but did not do anything neither

K6_PROMETHEUS_RW_HTTP_HEADERS=X-Scope-OrgID:
K6_PROMETHEUE_RW_HTTP_HEADERS_X-Scope-OrgID=
K6_PROMETHEUS_RW_HEADERS=X-Scope-OrgID:

I never get that invalid environment variable error.

Thanks
-Suleyman

8

Hi all,

Anybody have a solution for the above issue?

cc: @codebien

10

Hi @skutlu, Welcome to the community forum!

You can set the X-Scope-OrgID with the following environment variable:

K6_PROMETHEUS_RW_HTTP_HEADERS=X-Scope-OrgID:<tenant_id>
12

Hi @bandorko

That still does not seem to work… When I set it as you stated, I got authentication errors…

I will try to share my redacted config here in an hour or so. Meanwhile, if you have a working sample, I will also appreciate it

Thanks
-Suleyman

13

@skutlu:

I started mimir with docker compose as described here: Play with Mimir | Grafana Labs

I used this test script:

import http from 'k6/http';
import { sleep } from 'k6';

export default function () {
    http.get("https://google.com")
    console.log("iter")
    sleep(1)
}

And started it like this with k6 v0.48.0:

K6_PROMETHEUS_RW_HTTP_HEADERS=X-Scope-OrgID:demo K6_PROMETHEUS_RW_PASSWORD=admin K6_PROMETHEUS_RW_USERNAME=admin K6_PROMETHEUS_RW_SERVER_URL=http://localhost:9009/api/v1/push k6 run -o experimental-prometheus-rw script.js

If I run it without the K6_PROMETHEUS_RW_HTTP_HEADERS, I also get the error message:
Failed to send the time series data to the endpoint error=“got status code: 401 instead expected a 2xx successful status code” output=“Prometheus remote write”

1 Like
14

Hi @bandorko

I have a central LGTM stack running on K8s and from grafana-agents on all our clusters, I can see metrics written for different tenant ids, so multi-tenancy works on Mimir side.

Using same test script, I got following results.

If I use following command I get that “status code: 401” message in K6 run command output

K6_PROMETHEUS_RW_HTTP_HEADERS="X-Scope-OrgID:demo" K6_PROMETHEUS_RW_PASSWORD=<redacted> K6_PROMETHEUS_RW_USERNAME=<redacted> K6_PROMETHEUS_RW_SERVER_URL=https://prometheus.<redacted>.com/api/v1/push k6 run -o experimental-prometheus-rw test_script.js

If I use following, no errors and all sent to our Mimir… I can visualize with Grafana using a Datasource defined with X-Scope-OrgID:anonymous

K6_PROMETHEUS_RW_PASSWORD=<redacted> K6_PROMETHEUS_RW_USERNAME=<redacted> K6_PROMETHEUS_RW_SERVER_URL=https://prometheus.<redacted>.com/api/v1/push k6 run -o experimental-prometheus-rw test_script.js
15

Hey @skutlu,
what k6 version are you using?

I tried to re-run a test with K6_PROMETHEUS_RW_HTTP_HEADERS="X-Scope-OrgID:demo"as you did. I set as a remote write endpoint a mock server and I see the request’s header attached as expected in the form X-Scope-Orgid: demo. Can you try the same and see if the expected values are in the HTTP request, please?

I don’t know if it is relevant to your setup, but note that the case is different. Go changes the header key with a canonicalized format. If you are hitting directly Mimir then it shouldn’t be an issue as it shouldn’t care about it.

I hope it helps you.

1 Like
16

Hi @codebien

Happy holidays, at first :slight_smile:

I have tried with a Mock server on my laptop as below:

Started mock server with mockserver -serverPort 80 -logLevel DEBUG

Then run K6 with following CLI command:

K6_PROMETHEUS_RW_SERVER_URL="http://localhost" K6_PROMETHEUS_RW_USERNAME=mimir K6_PROMETHEUS_RW_PASSWORD="password" K6_PROMETHEUS_RW_HTTP_HEADERS="X-Scope-OrgID:demo" k6 run -o experimental-prometheus-rw test_script.js

          /\      |‾‾| /‾‾/   /‾‾/
     /\  /  \     |  |/  /   /  /
    /  \/    \    |     (   /   ‾‾\
   /          \   |  |\  \ |  (‾)  |
  / __________ \  |__| \__\ \_____/ .io

  execution: local
     script: test_script.js
     output: Prometheus remote write (http://localhost)

  scenarios: (100.00%) 1 scenario, 1 max VUs, 10m30s max duration (incl. graceful stop):
           * default: 1 iterations for each of 1 VUs (maxDuration: 10m0s, gracefulStop: 30s)

INFO[0001] iter                                          source=console
ERRO[0002] Failed to send the time series data to the endpoint  error="got status code: 404 instead expected a 2xx successful status code" output="Prometheus remote write"

     data_received..................: 33 kB  17 kB/s
     data_sent......................: 1.3 kB 668 B/s
...
... Redacted for sake of shorter message
...
     iterations.....................: 1      0.507468/s
     vus............................: 1      min=1      max=1
     vus_max........................: 1      min=1      max=1


running (00m02.0s), 0/1 VUs, 1 complete and 0 interrupted iterations
default ✓ [======================================] 1 VUs  00m02.0s/10m0s  1/1 iters, 1 per VU

On mockserver’s output I see:

2023-12-27 10:33:35 5.15.0 INFO 80 no expectation for:

  {
    "method" : "POST",
    "path" : "/",
    "headers" : {
      "content-encoding" : [ "snappy" ],
      "X-Scope-Orgid" : [ "demo" ],
      "X-Prometheus-Remote-Write-Version" : [ "0.1.0" ],
      "User-Agent" : [ "k6-prometheus-rw-output" ],
      "Host" : [ "localhost" ],
      "Content-Type" : [ "application/x-protobuf" ],
      "Content-Length" : [ "935" ],
      "Content-Encoding" : [ "snappy" ],
      "Accept-Encoding" : [ "gzip" ]
    },
    "keepAlive" : true,
    "secure" : false,
    "protocol" : "HTTP_1_1",
    "localAddress" : "[0:0:0:0:0:0:0:1]:80",
    "remoteAddress" : "[0:0:0:0:0:0:0:1]:53709",
    "body" : "Redacted for sake of shorter message"
  }

 returning response:

  {
    "statusCode" : 404,
    "reasonPhrase" : "Not Found"
  }

In the headers above, shouldn’t it be "X-Scope-Orgid" : "demo" (without square brackets)?

In Mimir logs for grafana-agent rw requests, I see without square brackets…

Thanks
-Suleyman

17

Hey @skutlu,
thanks for sharing the details. The header seems to be there as expected.

Can you try using a smaller environment like bandorko posted before Experimental-prometheus-rw output to a multi-tenant Mimir deployment - #13 by bandorko, please? If you find the same error then it would be good to share the docker compose with us so we can investigate a shared resource and reduce the level of guessing.

I assume it is the dump from your mock then it should just be an implementation detail of the loggers. HTTP allows to have multiple headers with the same key this is the reason why your mock is logging it as a list. In the raw HTTP request you shouldn’t see any bracket.

Happy New Year!

1 Like
18

Hi @codebien

Sorry for delay. I will try that and update here…

Thanks
-Suleyman

1 Like