V0.42 prometheus metrics missing

Hey folks, I’ve been playing around with the experimental-prometheus-rw feature and in my case I only see a subset of the k6 OSS metrics reported in the Grafana metrics viewer…

any thoughts on how I can push the remaining default k6_* metrics?

Hi @PlayStay

From the screenshot it seems you are sending the metrics to Cortex. Can you share the version? I would test this locally to see if I can reproduce the scenario. I’ve tested the prometheus write output with InfluxDb and it works, though there might be some incompatibilities with Cortex we can investigate.

Could you query directly your Cortex to get thet list of k6_* metrics? Are your results the same you see on Grafana when you filter for k6_? That is, you see reduced list of metrics?

Something similar to (localhost with no authentication):

curl -g 'http://localhost:9090/api/v1/series?' --data-urlencode 'match[]=up' --data-urlencode 'match[]={__name__=~".*k6_.*"}'

In my lab (local Prometheus v2.40.7 after running the es6sample.js test) I am getting:

{
    "status": "success",
    "data": [
        {
            "__name__": "k6_checks_rate",
            "check": "X-Test header is correct",
            "group": "::my group::json",
            "scenario": "default"
        },
        {
            "__name__": "k6_checks_rate",
            "check": "always fails",
            "group": "::my group::nested",
            "scenario": "default"
        },
        {
            "__name__": "k6_checks_rate",
            "check": "always passes",
            "group": "::my group::nested",
            "scenario": "default"
        },
        {
            "__name__": "k6_checks_rate",
            "check": "content type is html",
            "group": "::my group::html",
            "scenario": "default"
        },
        {
            "__name__": "k6_checks_rate",
            "check": "random value is \u003c 0.5",
            "group": "::my group",
            "scenario": "default"
        },
        {
            "__name__": "k6_checks_rate",
            "check": "status is 200",
            "group": "::my group::html",
            "scenario": "default"
        },
        {
            "__name__": "k6_checks_rate",
            "check": "status is 200",
            "group": "::my group::json",
            "scenario": "default"
        },
        {
            "__name__": "k6_checks_rate",
            "check": "top-level test",
            "scenario": "default"
        },
        {
            "__name__": "k6_checks_rate",
            "check": "welcome message is correct",
            "group": "::my group::html",
            "scenario": "default"
        },
        {
            "__name__": "k6_data_received_total",
            "scenario": "default"
        },
        {
            "__name__": "k6_data_sent_total",
            "scenario": "default"
        },
        {
            "__name__": "k6_group_duration_p99",
            "group": "::my group",
            "scenario": "default"
        },
        {
            "__name__": "k6_group_duration_p99",
            "group": "::my group::html",
            "scenario": "default"
        },
        {
            "__name__": "k6_group_duration_p99",
            "group": "::my group::json",
            "scenario": "default"
        },
        {
            "__name__": "k6_group_duration_p99",
            "group": "::my group::nested",
            "scenario": "default"
        },
        {
            "__name__": "k6_http_req_blocked_p99",
            "expected_response": "true",
            "group": "::my group::html",
            "method": "GET",
            "name": "http://test.k6.io/",
            "proto": "HTTP/1.1",
            "scenario": "default",
            "status": "308",
            "url": "http://test.k6.io/"
        },
        {
            "__name__": "k6_http_req_blocked_p99",
            "expected_response": "true",
            "group": "::my group::html",
            "method": "GET",
            "name": "https://test.k6.io/",
            "proto": "HTTP/1.1",
            "scenario": "default",
            "status": "200",
            "tls_version": "tls1.3",
            "url": "https://test.k6.io/"
        },
        {
            "__name__": "k6_http_req_blocked_p99",
            "expected_response": "true",
            "group": "::my group::json",
            "method": "GET",
            "name": "https://httpbin.org/get",
            "proto": "HTTP/2.0",
            "scenario": "default",
            "status": "200",
            "tls_version": "tls1.2",
            "url": "https://httpbin.org/get"
        },
        {
            "__name__": "k6_http_req_connecting_p99",
            "expected_response": "true",
            "group": "::my group::html",
            "method": "GET",
            "name": "http://test.k6.io/",
            "proto": "HTTP/1.1",
            "scenario": "default",
            "status": "308",
            "url": "http://test.k6.io/"
        },
        {
            "__name__": "k6_http_req_connecting_p99",
            "expected_response": "true",
            "group": "::my group::html",
            "method": "GET",
            "name": "https://test.k6.io/",
            "proto": "HTTP/1.1",
            "scenario": "default",
            "status": "200",
            "tls_version": "tls1.3",
            "url": "https://test.k6.io/"
        },
        {
            "__name__": "k6_http_req_connecting_p99",
            "expected_response": "true",
            "group": "::my group::json",
            "method": "GET",
            "name": "https://httpbin.org/get",
            "proto": "HTTP/2.0",
            "scenario": "default",
            "status": "200",
            "tls_version": "tls1.2",
            "url": "https://httpbin.org/get"
        },
        {
            "__name__": "k6_http_req_duration_p99",
            "expected_response": "true",
            "group": "::my group::html",
            "method": "GET",
            "name": "http://test.k6.io/",
            "proto": "HTTP/1.1",
            "scenario": "default",
            "status": "308",
            "url": "http://test.k6.io/"
        },
        {
            "__name__": "k6_http_req_duration_p99",
            "expected_response": "true",
            "group": "::my group::html",
            "method": "GET",
            "name": "https://test.k6.io/",
            "proto": "HTTP/1.1",
            "scenario": "default",
            "status": "200",
            "tls_version": "tls1.3",
            "url": "https://test.k6.io/"
        },
        {
            "__name__": "k6_http_req_duration_p99",
            "expected_response": "true",
            "group": "::my group::json",
            "method": "GET",
            "name": "https://httpbin.org/get",
            "proto": "HTTP/2.0",
            "scenario": "default",
            "status": "200",
            "tls_version": "tls1.2",
            "url": "https://httpbin.org/get"
        },
        {
            "__name__": "k6_http_req_failed_rate",
            "expected_response": "true",
            "group": "::my group::html",
            "method": "GET",
            "name": "http://test.k6.io/",
            "proto": "HTTP/1.1",
            "scenario": "default",
            "status": "308",
            "url": "http://test.k6.io/"
        },
        {
            "__name__": "k6_http_req_failed_rate",
            "expected_response": "true",
            "group": "::my group::html",
            "method": "GET",
            "name": "https://test.k6.io/",
            "proto": "HTTP/1.1",
            "scenario": "default",
            "status": "200",
            "tls_version": "tls1.3",
            "url": "https://test.k6.io/"
        },
        {
            "__name__": "k6_http_req_failed_rate",
            "expected_response": "true",
            "group": "::my group::json",
            "method": "GET",
            "name": "https://httpbin.org/get",
            "proto": "HTTP/2.0",
            "scenario": "default",
            "status": "200",
            "tls_version": "tls1.2",
            "url": "https://httpbin.org/get"
        },
        {
            "__name__": "k6_http_req_receiving_p99",
            "expected_response": "true",
            "group": "::my group::html",
            "method": "GET",
            "name": "http://test.k6.io/",
            "proto": "HTTP/1.1",
            "scenario": "default",
            "status": "308",
            "url": "http://test.k6.io/"
        },
        {
            "__name__": "k6_http_req_receiving_p99",
            "expected_response": "true",
            "group": "::my group::html",
            "method": "GET",
            "name": "https://test.k6.io/",
            "proto": "HTTP/1.1",
            "scenario": "default",
            "status": "200",
            "tls_version": "tls1.3",
            "url": "https://test.k6.io/"
        },
        {
            "__name__": "k6_http_req_receiving_p99",
            "expected_response": "true",
            "group": "::my group::json",
            "method": "GET",
            "name": "https://httpbin.org/get",
            "proto": "HTTP/2.0",
            "scenario": "default",
            "status": "200",
            "tls_version": "tls1.2",
            "url": "https://httpbin.org/get"
        },
        {
            "__name__": "k6_http_req_sending_p99",
            "expected_response": "true",
            "group": "::my group::html",
            "method": "GET",
            "name": "http://test.k6.io/",
            "proto": "HTTP/1.1",
            "scenario": "default",
            "status": "308",
            "url": "http://test.k6.io/"
        },
        {
            "__name__": "k6_http_req_sending_p99",
            "expected_response": "true",
            "group": "::my group::html",
            "method": "GET",
            "name": "https://test.k6.io/",
            "proto": "HTTP/1.1",
            "scenario": "default",
            "status": "200",
            "tls_version": "tls1.3",
            "url": "https://test.k6.io/"
        },
        {
            "__name__": "k6_http_req_sending_p99",
            "expected_response": "true",
            "group": "::my group::json",
            "method": "GET",
            "name": "https://httpbin.org/get",
            "proto": "HTTP/2.0",
            "scenario": "default",
            "status": "200",
            "tls_version": "tls1.2",
            "url": "https://httpbin.org/get"
        },
        {
            "__name__": "k6_http_req_tls_handshaking_p99",
            "expected_response": "true",
            "group": "::my group::html",
            "method": "GET",
            "name": "http://test.k6.io/",
            "proto": "HTTP/1.1",
            "scenario": "default",
            "status": "308",
            "url": "http://test.k6.io/"
        },
        {
            "__name__": "k6_http_req_tls_handshaking_p99",
            "expected_response": "true",
            "group": "::my group::html",
            "method": "GET",
            "name": "https://test.k6.io/",
            "proto": "HTTP/1.1",
            "scenario": "default",
            "status": "200",
            "tls_version": "tls1.3",
            "url": "https://test.k6.io/"
        },
        {
            "__name__": "k6_http_req_tls_handshaking_p99",
            "expected_response": "true",
            "group": "::my group::json",
            "method": "GET",
            "name": "https://httpbin.org/get",
            "proto": "HTTP/2.0",
            "scenario": "default",
            "status": "200",
            "tls_version": "tls1.2",
            "url": "https://httpbin.org/get"
        },
        {
            "__name__": "k6_http_req_waiting_p99",
            "expected_response": "true",
            "group": "::my group::html",
            "method": "GET",
            "name": "http://test.k6.io/",
            "proto": "HTTP/1.1",
            "scenario": "default",
            "status": "308",
            "url": "http://test.k6.io/"
        },
        {
            "__name__": "k6_http_req_waiting_p99",
            "expected_response": "true",
            "group": "::my group::html",
            "method": "GET",
            "name": "https://test.k6.io/",
            "proto": "HTTP/1.1",
            "scenario": "default",
            "status": "200",
            "tls_version": "tls1.3",
            "url": "https://test.k6.io/"
        },
        {
            "__name__": "k6_http_req_waiting_p99",
            "expected_response": "true",
            "group": "::my group::json",
            "method": "GET",
            "name": "https://httpbin.org/get",
            "proto": "HTTP/2.0",
            "scenario": "default",
            "status": "200",
            "tls_version": "tls1.2",
            "url": "https://httpbin.org/get"
        },
        {
            "__name__": "k6_http_reqs_total",
            "expected_response": "true",
            "group": "::my group::html",
            "method": "GET",
            "name": "http://test.k6.io/",
            "proto": "HTTP/1.1",
            "scenario": "default",
            "status": "308",
            "url": "http://test.k6.io/"
        },
        {
            "__name__": "k6_http_reqs_total",
            "expected_response": "true",
            "group": "::my group::html",
            "method": "GET",
            "name": "https://test.k6.io/",
            "proto": "HTTP/1.1",
            "scenario": "default",
            "status": "200",
            "tls_version": "tls1.3",
            "url": "https://test.k6.io/"
        },
        {
            "__name__": "k6_http_reqs_total",
            "expected_response": "true",
            "group": "::my group::json",
            "method": "GET",
            "name": "https://httpbin.org/get",
            "proto": "HTTP/2.0",
            "scenario": "default",
            "status": "200",
            "tls_version": "tls1.2",
            "url": "https://httpbin.org/get"
        },
        {
            "__name__": "k6_iteration_duration_p99",
            "scenario": "default"
        },
        {
            "__name__": "k6_iterations_total",
            "scenario": "default"
        },
        {
            "__name__": "k6_my_counter_total",
            "group": "::my group",
            "scenario": "default",
            "tag": "test"
        },
        {
            "__name__": "k6_my_rate_rate",
            "group": "::my group",
            "scenario": "default"
        },
        {
            "__name__": "k6_vus"
        },
        {
            "__name__": "k6_vus_max"
        },
        {
            "__name__": "up",
            "instance": "localhost:9090",
            "job": "prometheus"
        }
    ]
}

Which is what I then see in my browser.

If the metrics are not there in your Cortex either (we can discard any issues with Grafana querying), did you see any errors in the k6 logs when running the test related to writing the output? Is the test running successfully?

With this information I’m hoping we can keep digging into this.

Cheers!

Hi there @eyeveebe thank you for responding. Apologies for the tardy response. I’m using our corporate cortex appliance not a locally installed one. It appears (thanks to your example) I have a undiagnosable issue with my publicly acquired grafana dashboard JSON :slight_smile:

when I used the dashboard provided in the xk6-output-prometheus-remote/grafana/dashboards/dashboard-results.json file the k6_* key values were retrieved from the dashboard editor query pattern :man_shrugging:

so I have no idea how to reproduce the issue I saw with a properly setup dashboard json. json I did not write of course :triumph:

the dashboard I used to observe the lack of key values is likely from an ancient blog post - Combine k6 OSS and Prometheus for better observability I can’t say. However my problem appears to be gone.

However, I am having issues with the given dashboard in the that xk6 dashboard. it may not be appropriate for use given that I’m not using the xk6 extension in favor of the 0.42 experimental package. After manually updating the JSON from the xk6 repo dashboard-result.json most of the values display as null. Again after tinkering with the script I get most of the values to appear. Do you know of a dashboard template I can use as a shortcut for learning if not outright copying :).

Hey @PlayStay,

can you explain your current setup in more detail?

From what I understand, you’re running a k6 test on your local machine that’s pushing metrics to a Cortex instance (not plain Prometheus) hosted on your network. And you’re trying to visualize the metrics in a Grafana instance also hosted on your network, by importing the “k6 Test result” dashboard used in the xk6-output-prometheus-remote repository. Correct?

Let me first say that I also confirmed the dashboard works fine in the Docker Compose setup provided by xk6-output-prometheus-remote. I changed the k6 service to use the grafana/k6:0.42.0 image, and Prometheus received all k6 metrics, and the dashboard shows all results:

Given that you’re using a different data source, I think you just need to change the dashboard panels to point to your Cortex data source.

The dashboard uses a DS_PROMETHEUS variable to make this convenient, so you should be able to change it from the default prometheus-in-docker to your Cortex data source:
2023-01-25-121418_467x128_scrot

This is, of course, assuming that the k6 metrics are reaching Cortex, and from your screenshot it seems that some are.

I would start from scratch by recreating the dashboard just in case (without your manual changes to the dashboard JSON), and doing the data source change, and if it doesn’t work, then troubleshoot if metrics are in Cortex to begin with. I’m not familiar with Cortex to help you out there, but you should ask for help outside of a k6 forum. Preferably someone in your team or company would know best about how to configure this properly. From what I understand, Cortex is an API-compatible implementation of Prometheus, so k6 should work transparently with it.

Good luck!

1 Like

yes to all of your initial questions @imiric thanks for responding. also thanks for confirming the use of the dashboard-results template. I have or rather had made the changes to point my source to the cortex instance in my infra but still no go. I kinda figured I’d eventually have to recreate the dashboard from scratch but as you can probably guess I’m being lazy :wink: .However I’m trying to avoid creating from scratch because I’m Sure I’ll break the cool feature where the dashboard dynamically creates metrics panel automagically from the dashboard variable “testId”.

I’m not sure (yet) why I’m having issues with certain dashboard panels and functions such as irate vs rate not working as intended. :man_shrugging:

Thanks for the tips I’ll keep chopping until it my screen looks like yours. :mechanical_arm:

1 Like