How to minimize the size of the Output data

Hi,

I used the k6 load testing tool for executing the following load.

  • Transactions per second (TPS): 2500
  • Response payload size: 50 KB
  • Duration: 6 hours

I send the data to InfluxDB. After 6 hours the size of my DB was more than 4 GB. Is there any way in which I can reduce the data size?

Thanks,
Dilshan

Hi @Dilshan_Fernando, welcome to the community forum :tada:

I don’t have any exact and concrete things you must do to reduce the size but here are some … suggestions that I recommend you try out possibly with 10minute test and see what gives you the best output (you might need to use new influxdb instance as some of those change what things are tags and what are fields):

  1. reduce system-tags you probably don’t care about most of those but they are still both sent and saved and indexed. I would recommend you keep name(not url) , status, expected_response. The following method, error, group and scenario might be useful as well in case you are making use of them as well as check but if you can make it work with just expected_response you can drop your checks as well :wink:
  2. Move as many of the above that you won’t be needing as indexed to fields, this has significant implications for the size and the speed of ingestion although it’s dependant as well as on how much different values there are. You do that through listing tags that you have enabled but want as fields in the K6_INFLUXDB_TAGS_AS_FIELDS env variable.
  3. You can try to use telegraf to aggregate the metrics. This will definitely be the biggest improvement but also requires telegraf and settuping it. I recommend using the statsd input of telegraf as it has pretty good aggregation, but you will likely need to try stuff out. I have a comment with an example config
  4. Using telegraf you can also additionally ignore metrics you don’t care about while this is still not possible with pure k6.

Hope this helps you and please write back how it all went :slight_smile:

@mstoykov ,

Thanks for the details.

By using systemTags: ['status', 'error', 'error_code'], I was able to reduce some amount of data send by k6.

In the meantime, by sending k6 results to Datadog I was able to resolve this issue. With the Datadog, I didn’t see this issue.

Best Regards,
Dilshan