Running out of memory with k6 script on k8s pod

Grafana k6 OSS Support
1

I’m seeing an issue with my script terminating prematurely, and I assume it’s a memory problem.

I see the k6 ascii logo followed by “Killed”.

This pod has an allocated 2048 MiB’s (2GB) and the script is reading a 100 MB file into memory using SharedArray:

const hopEntries = new SharedArray('hops', function() {
    const dataArray = open(filePath).split('\n').filter(ua => ua.trim() !== '');
    return dataArray;
});

Is there a problem with the way I’m processing the CSV file? My understanding is this should create a SharedArray, which avoids loading the file in each VU. In my test scenario, I’ve preallocated 1 VU and set a max of 10 VU’s.

It looks like it dies before executing any of the code in the default function.

Some more info on this:

  • top output:
MiB Mem :  31637.2 total,    937.5 free,  14593.9 used,  16105.8 buff/cache
MiB Swap:      0.0 total,      0.0 free,      0.0 used.  16644.3 avail Mem

937.5 MiB free memory = 983 MB

The file is 100 MB.

Seems there is plenty of available memory.

3

Hi @nate.reed,
welcome to the community forum :tada:

Your assumption for the SharedArray is correct. However, if something else is not correct is difficult to say without reading the full code.

Can you post an anonymized version of your script, please? Better if you can you use something like https://httpbin.test.k6.io so we may also run it.

If it is complex code then you can try to run it with the –verbose and –paused mode to check where the run is getting into trouble reading the detailed logs.

4

Thanks, @codebien. Full script follows. Note I’m using the “ramp_up” scenario and have been playing with the VU settings to see if reducing them has any affect. To minimize memory requirements, I reduced the preallocated VU’s to 1.

It does seem to be terminating on the file load:

DEBU[0000] Loading...                                    moduleSpecifier="file:///tmp/k6-test-hop-attribution.js" originalModuleSpecifier=k6-test-hop-attribution.js
DEBU[0000] 'k6-test-hop-attribution.js' resolved to 'file:///tmp/k6-test-hop-attribution.js' and successfully loaded 3140 bytes! 
DEBU[0000] Gathering k6 runtime options...              
DEBU[0000] Initializing k6 runner for 'k6-test-hop-attribution.js' (file:///tmp/k6-test-hop-attribution.js)... 
DEBU[0000] Detecting test type for...                    test_path="file:///tmp/k6-test-hop-attribution.js"
DEBU[0000] Trying to load as a JS test...                test_path="file:///tmp/k6-test-hop-attribution.js"
DEBU[0000] Babel: Transformed                            t=298.848984ms
INFO[0000] Loading hops...                               source=console
Killed

Code:

import { check, group, sleep } from 'k6';
import http from 'k6/http';
import { Rate } from 'k6/metrics';
import { SharedArray } from 'k6/data';

const filePath = 'replayData/hops.log';

const hopEntries = new SharedArray('hops', function() {
    const dataArray = open(filePath).split('\n').filter(ua => ua.trim() !== '');
    return dataArray;
});

// Define options for the k6 test
let scenarios = {
    normal_usage: {
        executor: 'constant-arrival-rate',
        startTime: '0s',
        gracefulStop: '60s',
        duration: '30m',
        rate: 1,
        timeUnit: '1s',
        preAllocatedVUs: 10,
        maxVUs: 20
    },
    ramp_up: {
        preAllocatedVUs: 10,
        maxVUs: 100,
        startRate: 1,
        timeUnit: '1s',
        executor: 'ramping-arrival-rate',
        startTime: '0s',
        stages: [
            { target: 1, duration: '1m'},
            { target: 10, duration: '1m'},
            { target: 50, duration: '15m'},
            { target: 100, duration: '15m'},
            { target: 300, duration: '15m' },
            { target: 500, duration: '15m' }
        ]
    }
};

export const options = {
    discardResponseBodies: true,
    scenarios: {}
}

if (__ENV.scenario) {
    // Use just a single scenario if `--env scenario=whatever` is used
    options.scenarios[__ENV.scenario] = scenarios[__ENV.scenario];
    // Update the start time:

} else {
    // Use "normal"
    options.scenarios['normal_usage'] = scenarios['normal_usage'];
}

// Define a custom metric to track successful requests
const successfulRequests = new Rate('successful_requests');

// Use `--env mode=debug` (or `-e mode=debug`)
const debug = __ENV.mode && __ENV.mode.toLowerCase() === 'debug';

// Main test function
export default function () {
    // Pick a random User-Agent from the list
    const randomIndex = Math.floor(Math.random() * hopEntries.length);
    const hopEntry = hopEntries[randomIndex].replace(/"/g, '');

    if (debug) {
        console.log(hopEntry);
    }

    const fields = hopEntry.split('<!>');
    const affiliate = fields[1];
    const vendor = fields[2];
    const userAgent = fields[3];
    const qcookie = fields[4];

    const tid = Math.floor(Math.random() * 10000);
    const attributionUrl = `https://hop.clickbank-tst.net/?affiliate=${affiliate}&vendor=${vendor}&tid=${tid}`;

    const data = {
        headers: {
            'Content-Type': 'application/json',
            'User-Agent': userAgent,
            'Referrer': 'https://somereferrer.com?index=' + Math.floor(Math.random() * 100000)
        }
    };

    if (qcookie.length > 0) {
        data['headers']['Cookie'] = "q=" + qcookie;
    }

    // Use `--env debug`
    if (debug) {
        console.log(attributionUrl);
        console.log(data);
    }

    const res = http.get(attributionUrl, data);

    // Check for a successful response
    const checkRes = check(res, {
        'status is 200': (r) => r.status === 200,
    });

    // Record the successful request
    successfulRequests.add(checkRes);
}

Here are some examples from hops.log:

"1684182619170<!>anhealth<!>livpure<!>""Mozilla/5.0 (iPhone; CPU iPhone OS 16_3_1 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) Mobile/15E148 Instagram
 282.0.0.17.114 (iPhone11,6; iOS 16_3_1; en_US; en-US; scale=3.00; 1242x2688; 472241688) NW/3<!><!>"
"1684182619141<!>saritatips<!>trabajosoc<!>""Mozilla/5.0 (Linux; Android 12; SM-A042M Build/SP1A.210812.016; wv) AppleWebKit/537.36 (KHTML, like Gecko) Version/4
.0 Chrome/112.0.5615.135 Mobile Safari/537.36 musical_ly_2022903040 JsSdk/1.0 NetType/WIFI Channel/samsung_store AppName/musical_ly app_version/29.3.4 ByteLocale
/es ByteFullLocale/es Region/AR Spark/1.3.2.2-bugfix AppVersion/29.3.4 BytedanceWebview/d8a21c6<!><!>"
5

If I subset the file, I can get it to run. It would be helpful to understand what the memory requirements are so I know how to size this instance.

6

Also, another question: Is it advisable to read the entire file into memory? I know it must be possible to randomly access a row, but my JS/k6 skills are pretty basic. Might there be a recipe for doing this?

7

Hi @nate.reed :wave:,
sorry for the late reply.

I don’t see where you’re writing the Loading hops... log line in your script. In any case, did you run the script with a minimal version where you just load the SharedArray? Do you get a successful run?

I think the high memory usage could be affected by the fact you’re doing some heavy string manipulation in the default function.

Also, another question: Is it advisable to read the entire file into memory?

Reading the entire file is totally acceptable if you require it for your test and this is one of the cases where the SharedArray is useful. However, it is advisable to load a file with ready-to-use entry points instead of re-working them at runtime.

I hope it helps.

1 Like