we are in the very beginning of transitioning to k6 from Python performance tests at our company. This means we will use Python for a while, but going to start to collect metrics in CloudWatch(we HAVE to use AWS services). I know it’s a k6 forum and i will need this knowledge later when we can use k6 to run performance tests so please bear with me.
So continuing, some facts about CloudWatch custom metrics:
- CloudWatch stores the metric with sub-minute resolution down to one second
- if we have multiple data points per second we can publish statistic sets instead of single metrics, BUT it comes with limitations: besides the average, Cloudwatch can only store minimum and maximum for the pre-aggregated, 1 second based data sets
do i understand well, that if we choose metric collection for k6 in Cloudwatch(to me it’s not optional), we lose precision on some higher percentiles(p90, p95) of >1rps endpoint metrics? Because i presume that these higher percentiles will look more like p99, since as a result of the aggregation, there is a higher chance that the maximum of the 1 second based datasets will be more similar.
Is there any trick with the config params described in the k6 docs here to get as realistic results as possible?
I’m not a statistical expert, but i presume that the outcome very much depends on the number of samples(duration of the test) and the actual value of the number of (higher than 1 second) requests as well.