Ignore http calls made in Setup or Teardown in results?

Hey

Realised today that it appears any http calls that are made in the Setup or Teardown methods are included in the final results numbers, such as http_reqs.

Is there a way to disable this and only get reports on the main default function?

Cheers,
gy

If you mean the end-of-test summary that k6 emits, then unfortunately you can’t easily discard or ignore them… :disappointed: https://github.com/loadimpact/k6/issues/1321 is a proposal for a way to do that, but it’s not implemented yet. It (or something like it) is high on our priority list, but I can’t give you an ETA yet, sorry…

If you use an external output like InfluxDB/k6 cloud/etc., you can filter out setup() and teardown() metrics by their group or scenario tags.

Hmm, actually, you can use that and a few other tricks to filter out setup() and teardown() metrics in the end-of-test summary even now! Because you can (1) define thresholds on tags and (2) these threshold definitions generate sub-metrics, which are displayed in the end-of-test-summary, and (3) post-v0.27.0 you can apply tags in each k6 scenario! :tada: So, something like this:

import http from 'k6/http'

export let options = {
    vus: 5,
    duration: "10s",
    thresholds: {
        // We're using 'scenario:default' because that's the internal k6
        // scenario name when we haven't actually specified `options.scenarios`
        // explicitly and are just using the old execution shortcuts instead...
        'http_req_duration{scenario:default}': [
            // Some dummy threshold that's always going to pass. You don't even
            // need to have something here, I tried it and just this
            // `'http_req_duration{scenario:default}': []` is enough to trick
            // k6, but that's undefined behavior I can't promise we won't break
            // in the future...
            `max>=0`,
        ],
    },
}

export function setup() {
    http.get('https://httpbin.test.k6.io/delay/5?stage=setup')
}
export default function () {
    http.get('http://test.k6.io/?where=default')
    http.get('https://httpbin.test.k6.io/delay/1?stage=default')
}

export function teardown() {
    http.get('https://httpbin.test.k6.io/delay/7?stage=teardown')
}

will result in an end-of-test summary like this:

    data_received..............: 497 kB 21 kB/s
    data_sent..................: 11 kB  442 B/s
    http_req_blocked...........: avg=47.02ms  min=2.48µs   med=5.49µs   max=541.85ms p(90)=173.73ms p(95)=407.58ms
    http_req_connecting........: avg=19.31ms  min=0s       med=0s       max=132.97ms p(90)=131.37ms p(95)=132.16ms
    http_req_duration..........: avg=775.35ms min=136.49ms med=1.13s    max=7.13s    p(90)=1.13s    p(95)=1.14s   
    ✓ { scenario:default }.....: avg=641.27ms min=136.49ms med=644.4ms  max=1.15s    p(90)=1.13s    p(95)=1.14s   
    http_req_receiving.........: avg=3.76ms   min=61.65µs  med=896.96µs max=17.83ms  p(90)=11.89ms  p(95)=14.19ms 
    http_req_sending...........: avg=56.48µs  min=14.18µs  med=44.96µs  max=331.36µs p(90)=108.14µs p(95)=127.76µs
    http_req_tls_handshaking...: avg=24.36ms  min=0s       med=0s       max=341.86ms p(90)=0s       p(95)=276.07ms
    http_req_waiting...........: avg=771.52ms min=133.11ms med=1.13s    max=7.13s    p(90)=1.13s    p(95)=1.14s   
    http_reqs..................: 82     3.402872/s
    iteration_duration.........: avg=1.46s    min=1.27s    med=1.28s    max=5.67s    p(90)=1.87s    p(95)=1.87s   
    iterations.................: 40     1.659938/s
    vus........................: 0      min=0 max=5
    vus_max....................: 5      min=5 max=5

Notice how the ✓ { scenario:default } row doesn’t contain the long setup() and teardown() times. You can do this even if you have multiple scenarios, but then you’d have to add tags for each one. On the flip side, that gives you even more flexibility, you’d be able to get any number of cross-sections for each sub-metric.

See the last example from the “Advenced examples” section in the scenarios docs (sorry, can’t link directly to it), the one with:

export let options = {
  // ...
  thresholds: {
    // we can set different thresholds for the different scenarios because
    // of the extra metric tags we set!
    'http_req_duration{test_type:api}': ['p(95)<250', 'p(99)<350'],
    'http_req_duration{test_type:website}': ['p(99)<500'],
    // we can reference the scenario names as well
    'http_req_duration{scenario:my_api_test_2}': ['p(99)<300'],
  },
};

If you run that, the end of test summary will look somewhat like this:

...
    http_req_duration..............: avg=140.2ms  min=131.85ms med=136.63ms max=233.94ms p(90)=149.76ms p(95)=154.92ms
    ✓ { scenario:my_api_test_2 }...: avg=148.57ms min=139.63ms med=145.59ms max=233.94ms p(90)=157.21ms p(95)=167.33ms
    ✓ { test_type:api }............: avg=148.95ms min=139.63ms med=145.69ms max=233.94ms p(90)=158.46ms p(95)=168.87ms
    ✓ { test_type:website }........: avg=135.07ms min=131.85ms med=134.01ms max=160.74ms p(90)=139.29ms p(95)=140.97ms
...
2 Likes

Ah yeah interesting. Thanks as always @ned. The suggestions would need us to rework all of our tests, which we’re not against doing mind. We’ll consider this and keep an eye on https://github.com/loadimpact/k6/issues/1321

Sorry to hear that… :disappointed: IIRC, you had a wrapper tool that was parsing the k6 end-of-test summary - is this the reason for you having to rework the tests? If not, can you share the cause?

And if that is the cause, maybe the --summary-export k6 option will be of use for you? It allows you to output the end-of-test summary results in a JSON file. It will contain something like this for my first example above:

{
    "metrics": {
        // ...
        "http_req_duration{scenario:default}": {
            "avg": 639.2618575,
            "max": 1136.481917,
            "med": 639.2618575,
            "min": 142.041798,
            "p(90)": 1037.0379051000002,
            "p(95)": 1086.75991105,
            "thresholds": {
                "max>=0": false
            }
        },
        // ...
    }
}

Yeah that’s right, we parse the output to retrieve RPS achieved basically. We need standardised names to parse each test so this isn’t impossible since mentioned but would indeed mean some rework.

It’s not a major issue for us though, just sometimes we’ll see RPS achieved higher than the target as it includes those calls we make in setup and teardown.