How p(95) and p(90) works?

Grafana k6 OSS Support
1

Hello Community, I have been trying to understand how P(95) and P(90) work and will really appreciate your help for some of the queries.

Background: I have 1 scenario which runs 1 request with 1 Vu and 10 iteration using per-vu-iterations and I want to make sure that 8 out of 10 request are below 10ms

For a test scenario having 1 request with following option

scenarios: {
        GetApplets: {
            executor: 'per-vu-iterations',
            exec: 'getApplets',
            vus: 1,
            iterations: 10,
            maxDuration: '50s',
        },
    thresholds: {
        'http_req_duration{scenario:GetApplets}': ['p(80)<10'],

Output with response.timings.duration for each iteration in console log

k6 run .\systemtable.js

          /\      |‾‾| /‾‾/   /‾‾/
     /\  /  \     |  |/  /   /  /
    /  \/    \    |     (   /   ‾‾\
   /          \   |  |\  \ |  (‾)  |
  / __________ \  |__| \__\ \_____/ .io

  execution: local
     script: .\systemtable.js
     output: -

  scenarios: (100.00%) 1 scenario, 1 max VUs, 1m20s max duration (incl. graceful stop):
           * GetApplets: 10 iterations for each of 1 VUs (maxDuration: 50s, exec: getApplets, gracefulStop: 30s)

INFO[0008] GetApplets response.timings.duration 55.1737  source=console
INFO[0008] GetApplets response.timings.duration 4.5161   source=console
INFO[0008] GetApplets response.timings.duration 3.5661   source=console
INFO[0008] GetApplets response.timings.duration 4.8692   source=console
INFO[0008] GetApplets response.timings.duration 4.6393   source=console
INFO[0008] GetApplets response.timings.duration 5.1328   source=console
INFO[0008] GetApplets response.timings.duration 5.0173   source=console
INFO[0008] GetApplets response.timings.duration 4        source=console
INFO[0008] GetApplets response.timings.duration 3.5377   source=console
INFO[0008] GetApplets response.timings.duration 4.0129   source=console

running (0m00.7s), 0/1 VUs, 10 complete and 0 interrupted iterations
GetApplets ✓ [======================================] 1 VUs  00.3s/50s  10/10 iters, 10 per VU
     ✓ GetApplets Response Code is 200
     ✓ GetApplets Body not null

     █ setup

     checks.........................: 100.00% ✓ 20        ✗ 0
     data_received..................: 46 kB   69 kB/s
     data_sent......................: 19 kB   29 kB/s
     http_req_blocked...............: avg=111.87µs min=0s     med=0s      max=1.05ms   p(90)=422.56µs p(95)=943.18µs
     http_req_connecting............: avg=88.73µs  min=0s     med=0s      max=989.3µs  p(90)=418.72µs p(95)=527.71µs
     http_req_duration..............: avg=27.42ms  min=3.53ms med=14.89ms max=274.07ms p(90)=45.42ms  p(95)=54.29ms
       { expected_response:true }...: avg=27.42ms  min=3.53ms med=14.89ms max=274.07ms **p(90)=45.42ms  p(95)=54.29ms**
**     ✗ { scenario:GetApplets }......: avg=13.43ms  min=3.53ms med=14.25ms max=55.17ms**  **p(90)=20.95ms  p(95)=22.68ms**
     http_req_failed................: 0.00%   ✓ 0         ✗ 23
     http_req_receiving.............: avg=909.29µs min=0s     med=869.6µs max=6.96ms   p(90)=1.52ms   p(95)=1.55ms
     http_req_sending...............: avg=92.62µs  min=0s     med=0s      max=1.02ms   p(90)=431.38µs p(95)=548.04µs
     http_req_tls_handshaking.......: avg=0s       min=0s     med=0s      max=0s       p(90)=0s       p(95)=0s
     http_req_waiting...............: avg=26.41ms  min=3.51ms med=14.52ms max=267.1ms  p(90)=44.25ms  p(95)=53.26ms
     http_reqs......................: 23      34.843797/s
     iteration_duration.............: avg=59.53ms  min=20ms   med=24.04ms max=363.57ms p(90)=77.77ms  p(95)=220.67ms
     iterations.....................: 10      15.149477/sERRO[0009] some thresholds have failed

I looked at P function at Link
and also tried on various portals to calculate percentile using my values 3.5377, 3.5661, 4, 4.0129, 4.5161, 4.6393, 4.8692, 5.0173, 5.1328 ,55.1737 and below are results
p(90) : 30.15325ms
p(95) : 55.1737ms
p(80) : 5.07ms

But on the k6 summary I get following
p(90)=20.95ms p(95)=22.68ms

I want to validate that out of 10 request , 8 request have duration below 10ms, was wondering how can we achieve it ?

Any help will be appreciated. Thank you :slight_smile:

2

Hi @kishan,

The numbers you are showing seem off. I did calculate them with a basic numpy.percentile script such as :

>>> numpy.percentile([3.5377, 3.5661, 4, 4.0129, 4.5161, 4.6393, 4.8692, 5.0173, 5.1328 ,55.1737], [50.0, 80.0,85.0,87.5, 90.0,92.5, 95.0, 99.0], interpolation='linear')
array([ 4.5777   ,  5.0404   ,  5.092375 ,  5.1183625, 10.13689  ,
       21.3960925, 32.655295 , 50.670019 ])

This means that p(80) is 5.0404, p(90) is 10.13689 and p(95) is 32.655295 and then I confirmed that for the numbers you gave k6 will give those exact results.

I would expect that you have more than just that request in the scenario? You can try to tag the request with a specific tag and make your percentiles based on that.

I would argue that percentiles aren’t really all that well defined (and definitely not very stable) for small sample calculations. Also as it’s mentioned in different places and as numpy’s interpolation parameter suggests there are different ways to get around that fact which changes the values calculated.

(also the default numpy interpolation is linear which is also what k6 does which is why the numbers match, you probably used a tool using something different).

3

Yes seems like for small calculations it is not stable. I found a way to use Counters instead of P(80) function.
Also I saw there was an additional auth token request inside scenario which I resolve using Link

Thank you @mstoykov