Measuring web socket performance

k6 support
#1

Hi,

I’m considering the usage of k6 to test the performance of a web socket app. The scenario is as follows:
Client continuously sends a sequence of messages, say, A, B, C in a loop. After each message, the client waits for the response from the server and then sends another message.
I’d like to have a report of how long each interaction - A, B and C - takes, ideally with basic stuff (average, mean, percentiles) calculated.
By default, k6 only reports overall WS statistics. So, it seems to me I have to manually keep track of what message was sent and how much time had passed since it was sent.
Am I correct in my assumption? Is there a better way to measure the performance w/o manual timers?

#2

Hi @vlad_stryapko, welcome to the forum :tada:

As far as I am aware WebSockets have no notion of request-response pair as HTTP does. This means that while k6 can measure how long it takes for a response to be returned after the last time a user sends something, there is no way to know they are for the same thing. There is even nothing requiring that there will be a response to something someone sends or that there will a send a request for a received one. So in reality k6 has no way of knowing what it needs to measure.

So yes - you will need to that yourself and probably use a Trend.

I don’t know if any other tool manages to do something about it, but if you have any ideas you can open an issue.

I would like to also tell you that the current WebSocket API has … known deficiencies, the major one being having its own async loop which is something that will need to go and a completely new API will come in its place (the old one will be deprecated but available for quite some time). This though is unlikely to happen in the next few releases as there still no global async loop in k6 and there are other higher priority things the k6 team will like to concrete on. I am mostly saying this so you know that the current state of the API and the functionality is … will change, hopefully for the better ;).

Hope this helps you and Good luck

#3

Thanks for the reply and also for the additional info about the current WS API situation, it’s very helpful. One small thing I’d like to clarify: is it correct that k6 doesn’t have a precise time-measurement API? I’d like not to use Date since in my experience it’s usually not the best choice.

I don’t have any ideas in particular, though I can share how it’s currently done with JMeter: it has a plugin which can read from the socket. In the JMeter test itself we have a simple while loop which sends a message with a generated UUID and then reads the data from the socket until it encounters a response with the same UUID. It’s needed because, as you mentioned, the server might send a message which is absolutely unrelated to the actual request we sent before. So, the loop is exited if and only if we managed to read the response to the request. JMeter itself handles the measurement of the time passed and then a separate plugin is responsible for calculating the statistics.

#4

is it correct that k6 doesn’t have a precise time-measurement API? I’d like not to use Date since in my experience it’s usually not the best choice.

Yes, someone asked this recently here Process.hrtime cannot be access inside js file. Given the apparent interest, maybe we should add it … but the API is under question(as nodejs has one and browsers have different one). I also … highly doubt that there will be a measurable difference in the majority of the cases between using it and Date.now()

The thing about JMeter sounds like you’ve scripted it(aka what you are asking if you need to do here) … and I still don’t understand how Jmeter figures out what it needs to measure … does it measure how long the loop takes to execute?

#5

Cool, thanks. I suppose I’ll either use Date or performance.now from Node. I don’t care that much about nanoseconds, but I definitely care about milliseconds.

Yep, it does. The main difference from k6 WS is that from JMeter’s POV WS interaction is synchronous and not event-driven so it’s easy to measure. We still have to include the workaround with correlation UUID, though.
If k6 had the same API, it would look like this:

group('Message A', function () {
      var id = UUID.new()
      socket.send(new MessageA(id))
      var incomingId = null
      while (incomingId != id)
      {
          const msg = socket.read()
          incoming = msg.id
      }
    });

So, you leave that group as soon as you 1. sent message A and 2. received a response. In my particular case, you’d also have groups for messages B and C. We also have a general while loop because we obviously want to send the messages multiple times, but in k6 a default function would suffice.