Maximum number of tests per month is rather low

I am evaluating k6 for automated performance regression testing of a product I work on. It looks pretty good, but I ran into some quirks.

The first problem that I ran into is that we have dozens of tests that need to be run sequentially, with detailed histogram metrics aggregated for each individual test.

Scenarios are mentioned as a solution, but they have two issues:

  • Scenarios don’t seem adequately cover this type of execution right now, not just because it’s impossible to specify sequential ordering (without using startTime)
  • Also, the cloud UI doesn’t appear to not distinguish between scenarios at all. In other words, scenarios aren’t “test runs”, they’re just phases of a test.

So the only option appears to be to run one k6 test per test.

The second problem that I ran into is that the pricing model appears to assume you run relatively few test runs. With dozens of tests that each run on every CI push, we would exceed the limit very quickly. We could reasonably compromise here and run the tests just once a day per active branch, but with dozens of tests, we’d still exceed that limit. For example, assume 20 tests and that our product is being actively worked on daily. That’s perhaps 5,000 runs per year, more than twice the limit for Team/Pro plans. (Custom/Enterprise is not on the table, sorry!) Maybe we could dial it down further and only run tests for each pull request and mainline merge, but it seems a bit silly to be so arbitrarily restrictive.

Ideally, I’d like to go into the performance results for any given GitHub commit and compare it with a different GitHub commit.

Another thing I am curious about is that the only result view offered is a timeline. Seeing request rates over time can be useful, I guess, and we do want to store “trends” like GC pause times where a timeline can be a nice visualization. But a 1-100th percentile latency histogram would be much more useful, and tells you real things about your data.

Hey @atombender!

My name is Mark and I’m the Head of Customer(CS and Sales) here at k6 cloud. I’m happy you asked this in the public forum, as this question has been coming up more often and I don’t think our pricing page truly displays our flexibility. :slight_smile:

These test are you running, it sounds like they need to be in their own test run and having their own result data set, is that correct? If this is the case you should run separate tests for each.

The pricing model you see on our pricing page is designed to be as self service as possible based on our past experience of what users typically asked for. Simply, those tiers were designed to be one-size-fits-most, competitive, and not require a salesperson.

We offer custom or enterprise plans for those that have more unique requirements. I’ll strongly note that not all custom plans are “enterprise” (but all enterprise plans are indeed custom). Maybe we should make this clearer on the pricing page.

At this point, I think it would be best to talk directly about your testing requirements and we can suggest some things based off of what you tell us. Can you send us a quick note and I’ll make sure to reply right away: Sales (at) k6 (dot) io

Final note for you and anyone else reading, it’s not uncommon for us to create “tiered” plans with different requirements of VUs/duration etc. all in a single plan. e.g. a CI tier, a stress testing tier, etc.

If there’s more question from you or anyone else, please ask. We are happy to help.

Thanks, sounds like you might be able to accommodate us. I will reach out by email.