The flush operation took higher than the expected set push interval

Hi,

I am using K6 in a Docker with influx db and Grafana dashboard.

I am using rampingArrivalRate as shown below.

When I run the tests, in the containers I see lots of the following warnings. Anyone know what this means and how I can fix it please?

time="2021-12-02T10:48:56Z" level=warning msg="The flush operation took higher than the expected set push interval. If you see this message multiple times then the setup or configuration need to be adjusted to achieve a sustainable rate." output=InfluxDBv1 t=1.67979745s

Here are the options I am using in the test:

export let options = {
  scenarios: {
    successfulLogin_rampingArrivalRate: {
      executor: 'ramping-arrival-rate',
      startRate: 280,
      timeUnit: '5m',
      preAllocatedVUs: 2,
      maxVUs: 8,
      stages: [
        { target: 280, duration: '1m' }, //Ramp up
        { target: 280, duration: '5m' }, //Stay at same level
        { target: 0, duration: '30s' }, // Ramp down
      ],
    },
  },

Hi @MattK6,
the requests for flushing metrics to InfluxDB are taking longer (1.6s in your reported case) of the set flush interval (default to 1s) so it means that k6 is probably flushing more metrics than InfluxDB can really handle.
Do you see a high value of CPU and/or RAM for your InfluxDB instance? Are you using the same machine for running k6 and InfluxDB or are they separated?

I list here some steps to try for getting a better response time from InfluxDBv1:

  • You could set only the SystemTags that you really need for your analysis. This can help for reducing the number of metrics that k6 is generating and InfluxDB have to ingest.
  • Tune the options of the k6 output for InfluxDB:
    • You could consider excluding some tags to be indexed K6_INFLUXDB_TAGS_AS_FIELDS that should speed up the ingestion phase for InfluxDB. Note that not indexed tags could be slow when you need to query them.
    • Reduce the K6_INFLUXDB_PUSH_INTERVAL and increase the K6_INFLUXDB_CONCURRENT_WRITES options for flushing batches with a smaller number of metrics. You probably will need to check the logs with the –verbose option for setting the right values.
  • Try to use Telegraf for aggregating some metrics before sending them to InfluxDB.

If none of the previous suggestions works for you then you could try the new InfluxDBv2 extension and see if it’s able to ingest your metrics faster.

Hey man!

Would you please be so kind and explain (an example) how do manipulate these:

K6_INFLUXDB_PUSH_INTERVAL
K6_INFLUXDB_CONCURRENT_WRITES

Is it via the command line? Like that:

$ k6 run K6_INFLUXDB_PUSH_INTERVAL=blah script.js

Hi @fatkobra,
you can find documentation about how to use the options at the doc page Using options.

So, as the example mention, in the case of shell you can use:

$ K6_INFLUXDB_PUSH_INTERVAL=1s k6 run script.js

Thank you so much @codebien for your input.

I was able to successfully use K6_INFLUXDB_PUSH_INTERVAL=2s and see the positive changes.

K6_INFLUXDB_PUSH_INTERVAL=2s k6 run --out influxdb=https://influxdb.blah.blahblah:portNo/somedb script.js --quiet