I try to do heavy load test with target more than 63000 VU using VM as worker.
I get error about limitation file size JSON from k6 output.
Here example log
Test ✓ [ 100% ] 63000 VUs 3m0s 6299754/6300000 iters, 100 per VU
time="2023-01-10T04:58:00Z" level=error msg="Sample couldn't be marshalled to JSON" error="write target/reports/report.json: file too large" filename=target/reports/report.json output=json
time="2023-01-10T04:58:02Z" level=info msg="[k6-reporter v2.3.0] Generating HTML summary report" source=console
time="2023-01-10T04:58:28Z" level=error msg="Sample couldn't be marshalled to JSON" error="write target/reports/report.json: file too large" filename=target/reports/report.json output=json
Any suggestion for manipulate output?
Welcome to the community forums
I imagine you are exporting all the metrics, the granular data points, in JSON format with the JSON output. Please clarify if me if I’m wrong. This can create a big file with a large test. Especially when using JSON, since it’s verbose.
One solution is using k6’s built-in support for outputting the results in other formats, like Prometheus, InfluxDB, or maybe even CSV (less verbose than JSON). These formats are more suitable for large data sets. Especially streaming the results to a data store.
If you want to see only the aggregated data, you can export the end-of-test summary to a JSON file. For more details, refer to the
handleSummary() topic in the end-of-test summary docs.
There is probably also the option to increase the maximum file size limit allowed by your system. However, this may not be feasible or desirable. Or reducing the amount of data that is being recorded by k6.
I hope this helps.
Thank you @eyeveebe for insight.
I will try again, and back with result.
I am back.
So csv still cannot solve my problem.
So. I fix problem to do something in my VM. I set sysctl file size limit to
/etc/security/limits.conf than add this line
k6-user hard fsize -1
Then restart vm.
For 60K VU, k6 generate JSON output with size 26GB.
And then move to solve problem how generate report from that JSON file, work on progress.