How to load test list of URLs

karim · January 12, 2023, 12:24am

Hello,

I’m trying to load test a Drupal site that has ~1000 pages. I have a CSV of the page URLs exported via the sitemap, and I would like to create a test that randomly hits of the URLs at each user request.

I found this article that seems to explain exactly what I’m trying to achieve Data Parameterization but I think it refers to an older version of k6, because I cannot find any mention of a “Load Impact Data Store” in the current docs or UI.

I would appreciate some help about that.

Thanks!
Karim

eyeveebee · January 12, 2023, 2:40pm

Hi @karim

Welcome to the community forums

And many thanks for spotting this. The blog is indeed for old k6 versions and we’ve made a note to review or archive it.

You can find updated documentation here:

SharedArray to read a file and share between VUs.
jslib randomItem(array) to get a random element from an array of URLs.

With this, you can build a simple script that as starting point like the one below:

import exec from 'k6/execution';
import http from 'k6/http';
import { SharedArray } from 'k6/data';
import { sleep } from 'k6';
import papaparse from 'https://jslib.k6.io/papaparse/5.1.1/index.js';
import { randomIntBetween } from "https://jslib.k6.io/k6-utils/1.0.0/index.js";
import { randomItem } from 'https://jslib.k6.io/k6-utils/1.2.0/index.js';

const sharedData = new SharedArray("URLs", function () {
    let data = papaparse.parse(open('urls.csv'), { header: true }).data;
    return data;
});

export const options = {
    vus: 2,
    duration: '30s',
};

export default function () {
    const url = randomItem(sharedData).url;
    console.log(`VU: ${exec.vu.idInTest}, iteration: ${exec.scenario.iterationInTest}], url: ${url} Starting iteration...`);
    http.get(url);
    sleep(randomIntBetween(1, 3));
}

Which relies on a CSV file urls.csv with the following content:

url
https://httpbin.test.k6.io/product/01
https://httpbin.test.k6.io/product/02
https://httpbin.test.k6.io/product/03
https://httpbin.test.k6.io/product/04
https://httpbin.test.k6.io/product/05
https://httpbin.test.k6.io/product/06
https://httpbin.test.k6.io/product/07
https://httpbin.test.k6.io/product/08
https://httpbin.test.k6.io/product/09
https://httpbin.test.k6.io/product/10

When running the script k6 run test.js, you should see it randomly hits URLs:


          /\      |‾‾| /‾‾/   /‾‾/   
     /\  /  \     |  |/  /   /  /    
    /  \/    \    |     (   /   ‾‾\  
   /          \   |  |\  \ |  (‾)  | 
  / __________ \  |__| \__\ \_____/ .io

  execution: local
     script: test.js
     output: -

  scenarios: (100.00%) 1 scenario, 2 max VUs, 1m0s max duration (incl. graceful stop):
           * default: 2 looping VUs for 30s (gracefulStop: 30s)

INFO[0000] VU: 2, iteration: 0], url: https://httpbin.test.k6.io/product/04 Starting iteration...  source=console
INFO[0000] VU: 1, iteration: 1], url: https://httpbin.test.k6.io/product/05 Starting iteration...  source=console
INFO[0002] VU: 2, iteration: 2], url: https://httpbin.test.k6.io/product/01 Starting iteration...  source=console
INFO[0002] VU: 1, iteration: 3], url: https://httpbin.test.k6.io/product/07 Starting iteration...  source=console
INFO[0005] VU: 2, iteration: 4], url: https://httpbin.test.k6.io/product/09 Starting iteration...  source=console
INFO[0006] VU: 1, iteration: 5], url: https://httpbin.test.k6.io/product/07 Starting iteration...  source=console
INFO[0007] VU: 2, iteration: 6], url: https://httpbin.test.k6.io/product/08 Starting iteration...  source=console
INFO[0007] VU: 1, iteration: 7], url: https://httpbin.test.k6.io/product/07 Starting iteration...  source=console
INFO[0008] VU: 2, iteration: 8], url: https://httpbin.test.k6.io/product/02 Starting iteration...  source=console
INFO[0010] VU: 1, iteration: 9], url: https://httpbin.test.k6.io/product/03 Starting iteration...  source=console
INFO[0010] VU: 2, iteration: 10], url: https://httpbin.test.k6.io/product/03 Starting iteration...  source=console
INFO[0012] VU: 1, iteration: 11], url: https://httpbin.test.k6.io/product/02 Starting iteration...  source=console
INFO[0013] VU: 2, iteration: 12], url: https://httpbin.test.k6.io/product/06 Starting iteration...  source=console
INFO[0013] VU: 1, iteration: 13], url: https://httpbin.test.k6.io/product/06 Starting iteration...  source=console
INFO[0014] VU: 2, iteration: 14], url: https://httpbin.test.k6.io/product/01 Starting iteration...  source=console
INFO[0015] VU: 2, iteration: 15], url: https://httpbin.test.k6.io/product/09 Starting iteration...  source=console
INFO[0016] VU: 1, iteration: 16], url: https://httpbin.test.k6.io/product/01 Starting iteration...  source=console
INFO[0016] VU: 2, iteration: 17], url: https://httpbin.test.k6.io/product/09 Starting iteration...  source=console
INFO[0017] VU: 1, iteration: 18], url: https://httpbin.test.k6.io/product/06 Starting iteration...  source=console
INFO[0017] VU: 2, iteration: 19], url: https://httpbin.test.k6.io/product/10 Starting iteration...  source=console
INFO[0018] VU: 1, iteration: 20], url: https://httpbin.test.k6.io/product/06 Starting iteration...  source=console
INFO[0019] VU: 2, iteration: 21], url: https://httpbin.test.k6.io/product/10 Starting iteration...  source=console
INFO[0020] VU: 1, iteration: 22], url: https://httpbin.test.k6.io/product/03 Starting iteration...  source=console
INFO[0021] VU: 2, iteration: 23], url: https://httpbin.test.k6.io/product/03 Starting iteration...  source=console
INFO[0022] VU: 1, iteration: 24], url: https://httpbin.test.k6.io/product/07 Starting iteration...  source=console
INFO[0024] VU: 2, iteration: 25], url: https://httpbin.test.k6.io/product/04 Starting iteration...  source=console
INFO[0025] VU: 1, iteration: 26], url: https://httpbin.test.k6.io/product/10 Starting iteration...  source=console
INFO[0025] VU: 2, iteration: 27], url: https://httpbin.test.k6.io/product/01 Starting iteration...  source=console
INFO[0028] VU: 1, iteration: 28], url: https://httpbin.test.k6.io/product/05 Starting iteration...  source=console
INFO[0028] VU: 2, iteration: 29], url: https://httpbin.test.k6.io/product/10 Starting iteration...  source=console
INFO[0029] VU: 1, iteration: 30], url: https://httpbin.test.k6.io/product/04 Starting iteration...  source=console
INFO[0029] VU: 2, iteration: 31], url: https://httpbin.test.k6.io/product/04 Starting iteration...  source=console

running (0m30.9s), 0/2 VUs, 32 complete and 0 interrupted iterations
default ✓ [======================================] 2 VUs  30s

     data_received..............: 26 kB   833 B/s
     data_sent..................: 4.4 kB  143 B/s
     http_req_blocked...........: avg=17.96ms  min=5µs      med=12µs     max=289ms    p(90)=14.9µs   p(95)=128.55ms
     http_req_connecting........: avg=6.29ms   min=0s       med=0s       max=100.78ms p(90)=0s       p(95)=45.31ms 
     http_req_duration..........: avg=105.58ms min=101.33ms med=104.56ms max=114.48ms p(90)=111.38ms p(95)=112.3ms 
     http_req_failed............: 100.00% ✓ 32       ✗ 0  
     http_req_receiving.........: avg=156.65µs min=36µs     med=163µs    max=357µs    p(90)=214.7µs  p(95)=243.99µs
     http_req_sending...........: avg=41.65µs  min=10µs     med=46.5µs   max=66µs     p(90)=56.9µs   p(95)=59.8µs  
     http_req_tls_handshaking...: avg=6.86ms   min=0s       med=0s       max=111.54ms p(90)=0s       p(95)=48.71ms 
     http_req_waiting...........: avg=105.38ms min=101.13ms med=104.36ms max=114.19ms p(90)=111.12ms p(95)=112.08ms
     http_reqs..................: 32      1.034946/s
     iteration_duration.........: avg=1.9s     min=1.1s     med=2.1s     max=3.11s    p(90)=3.1s     p(95)=3.1s    
     iterations.................: 32      1.034946/s
     vus........................: 2       min=2      max=2
     vus_max....................: 2       min=2      max=2

I hope this helps.

Cheers!

karim · January 12, 2023, 5:52pm

That’s fantastic, thanks! I’ll try this script and feedback here.

scottprive · April 10, 2024, 10:05pm

Hi, this question and answer is excellent; thank you both for pointing out how k6 can work with a CSV of URLs.

PROBLEM -
This errors, if you shorten urls.csv to fewer items:

error="Get \"\": unsupported protocol scheme \"\""

$ cat urls.csv
url
http://example.com
http://neverssl.com

$ k6 run test.js

          /\      |‾‾| /‾‾/   /‾‾/
     /\  /  \     |  |/  /   /  /
    /  \/    \    |     (   /   ‾‾\
   /          \   |  |\  \ |  (‾)  |
  / __________ \  |__| \__\ \_____/ .io

     execution: local
        script: test.js
        output: -

     scenarios: (100.00%) 1 scenario, 2 max VUs, 1m0s max duration (incl. graceful stop):
              * default: 2 looping VUs for 30s (gracefulStop: 30s)

INFO[0001] VU: 2, iteration: 0], url: http://neverssl.com Starting iteration...  source=console
INFO[0001] VU: 1, iteration: 1], url:  Starting iteration...  source=console
WARN[0001] Request Failed                                error="Get \"\": unsupported protocol scheme \"\""
INFO[0004] VU: 1, iteration: 2], url:  Starting iteration...  source=console
WARN[0004] Request Failed                                error="Get \"\": unsupported protocol scheme \"\""
INFO[0005] VU: 2, iteration: 3], url: http://example.com Starting iteration...  source=console
INFO[0005] VU: 1, iteration: 4], url:  Starting iteration...  source=console
WARN[0005] Request Failed                                error="Get \"\": unsupported protocol scheme \"\""
INFO[0007] VU: 1, iteration: 5], url: http://example.com Starting iteration...  source=console
INFO[0008] VU: 2, iteration: 6], url:  Starting iteration...  source=console
WARN[0008] Request Failed                                error="Get \"\": unsupported protocol scheme \"\""
INFO[0010] VU: 2, iteration: 7], url: http://neverssl.com Starting iteration...  source=console
INFO[0010] VU: 1, iteration: 8], url:  Starting iteration...  source=console
WARN[0010] Request Failed                                error="Get \"\": unsupported protocol scheme \"\""
INFO[0011] VU: 2, iteration: 9], url: http://example.com Starting iteration...  source=console
INFO[0012] VU: 2, iteration: 10], url: http://example.com Starting iteration...  source=console
INFO[0012] VU: 1, iteration: 11], url:  Starting iteration...  source=console
... pattern repeats...

Can I ask what you think is going on and how to resolve it?

if I drop the vus to just 1, and 2 urls, I get the same errors at about the same frequency.
If I duplicate my 2-url set (4 total lines) the error goes away, unless…
…if I increase vus to 5 (with the 4 urls), the problem returns.

So that’s one workaround… ensure the CSV file has more URLs than you run users, plus padding.

Is there some kind of “locking” of URL data to a request, so that another request (of that URL) can not begin, until the prior request is fully torn down? Any way to override that?

Thanks