Debugging HTTP requests

I created a performance test, and some of my http requests are failing. I can’t figure out for the life of me why its failing and the tool does not give any info. I tried doing the --http-debug and it doesn’t seem to help. I want to see the full request and full response but neither is provided, and I am unable to reproduce this issue in Postman. How can I debug this and see where the issue is?

Hi,

Could you please share some more context? Test script and terminal output would be helpful.
Running the test with --http-debug should indeed output both the request and response.

I take it you’ve used the postman-to-k6 converter to generate your test script?

Best regards,
Simme

Edit: --http-debug=full is required to show the body as well, as it defaults to excluding it.

I’m not sure how to share without sharing my entire performance test, and I can’t share it publicly. Can I email to you?

Sure, I’ll PM you my email

I sent email. I did not use postman-to-k6, I hand-crafted the whole script.

@mohamed_hava you need to use --http-debug=full if you want to see both the response and request, otherwise it just shows the headers, which is a lot more compact and in a lot of cases what people need.

      --http-debug string[="headers"]       log all HTTP requests and responses. Excludes body by default. To include body use '--http-debug=full'

OK, will give that a try. Thank you.

Great! Let me know if you get stuck and I’ll have a look at the files you sent. :+1:

Ok, I tried and its not helpful at all.

Request:
POST /hava/public/v1/driver/arrived HTTP/1.1
Host: perfapp.hava.bz
User-Agent: k6/0.26.1 (https://k6.io/)
Content-Length: 98
Accept: application/json
Content-Type: application/x-www-form-urlencoded
X-App-Version: hava.artillery.tests-1.1.0
Accept-Encoding: gzip

id=11&request_id=111&token=XXX

WARN[0057] Request Failed error="Post https://perfapp.hava.bz/hava/public/v1/driver/arrived: EOF"
INFO[0057] ERROR: https://perfapp.hava.bz/hava/public/v1/driver/arrived
INFO[0057] GoError: invalid response type
INFO[0057] BODY: null

There is no body even. So I’m not sure what’s going on. Please help.

id=11&request_id=111&token=XXX

Is your request body … .I guess XXX was something longer as the Content-Length is 98 (the header above).

After that given that the error is EOF I would guess the server just closes the connection for some reason without actually returning anything :smiley: .

You could try to use curl to make the request and with -v see what happens and maybe if it works with curl it is something with k6 … but currently this seems mostly like the server doesn’t like the request for some reason and just decides to close the connection.

I did the same thing in Postman and it was fine. I really can’t figure out why k6 is returning null when the server actually returns a value.

I replaced with 11 and XXX, there’s an actual user id and actual token.

My previous explanation still stands - your server closes the connection. Given the current code if the server returned a response we would’ve logged something or at least a message with FATAL. Unfortunately, as you are using https you can’t just use wireshark to check.

Maybe the server cares about the order in which the parameters are, and k6 currently doesn’t keep this [order(Order is not enforced when making a multipart/form-data request · Issue #1382 · grafana/k6 · GitHub). Can you try to handcraft your body - you will mostly need to order the parameters in the right order?

Also if your server expects that this will be multipart/form-data and not application/x-www-form-urlencoded ( I don’t know what postman does by default) , k6 doesn’t do multi-part/form-data by default.

And currently, I would argue it is best to make it manually instead of trying some of the magic ways that will make k6 do a multipart/form-data.

If you can make the request with postman maybe it can dump you exactly the headers and body and you can compare that yo what you see in the k6 output.

Will try that and get back to you…

I’ve been investigating this and battling with this all day with no solution. I tried postman, and powershell http request, and curl, and everything worked except for k6. Then I tried to install fiddler as some kind of proxy/middleman and restarted my computer, though I couldn’t get fiddler to work with k6, somehow after the restart it mysteriously started working again. No explanation given at all.

@mstoykov - there are no errors on the server logs either. This was a strange k6 issue (maybe go, maybe windows, maybe your node env) … just to circle back, everything looked good and the same request worked fine in postman. Additionally, there are about 42 other requests that work, but 2 out of the 42 had this strange error. I could not figure it out and tried everything. No output, no response, just EOF. The server did not show any errors or any issues (and I am heavily logging everything in the server) - the request never made it to the server. It was as if it was completely skipped over. I isolated the two failing methods, and they worked just fine, which led me to believe maybe the call right before had an issue, so I commented that out and still I had the same issue. After spending all day on it, all it took was a computer restart to fix everything.

Anwyays thanks for the support, now it works and I’ll continue working on the tests.

Ok, I celebrated too early. The error is back now, and I am hitting a brick wall. I have no idea why it is failing again now.

What causes k6 to report EOF with the invalid response type?

EOF is the error we get from the golang implementation - AFAIK it just means that … we got EOF when we didn’t expect it - so the connection was closed from the other end.

The invalid response type is because you called response.json() on a response with a null body or at least I don’t know from what else it could be.

Does this happen if you only run 1 iteration 1 Vu? or only if you have multiple iterations/VUs?

While I don’t find it unlikely that the golang implementation has issues on windows, I somewhat doubt it, as we do get users on windows and … well, people use golang on windows a lot - so something like this would’ve probably been easy to find and already fixed. You could try running it in docker, I have heard from users that k6 works wonderfully even on windows :).

This still could be the unordered nature of golang’s maps, so if you haven’t tried it - try the custom build body - with only one request and move the id,token,request_id params around. It is very likely that they will be in the “correct” order some of the time and then not the rest.

Apart from that … this smells more and more of an antivirus/firewall issue where something decides that some request is not okay and is going to stop it before getting to the server at all. Maybe there is something that is rate-limiting and k6 does request faster (or if you’re using more VUs in parallel) than whatever else you used to test it.

noConnectionReuse will make it so you always make a new connection if it happens to be an issue with reusing and old one, although this still seems like a rate limit thing … maybe a bad implementation of some server of limiting the number of request on a single http2 connection.

And the only other thing I can think of telling you is to is try wireshark, even without key logging in k6 you will still be able to see where traffic goes and when packages get fired and connections closed. You would need to make certain there are … fewer things doing networking on your machine as otherwise, it will way too noisy

Sorry, but I am starting to run out of good and easy to test ideas :frowning: , hopefully, one of the above will shine some light on the mystery

I saw that the used k6 version was v0.26.1, and it has known issues on Windows. Not these specific ones, but it’s probably worth downloading v0.26.2 and try again?

1 Like

Update- so I’m not sure what was causing the error, maybe it was a server side issue or something… but I was only able to reproduce it during a load. It might’ve been a timeout, but there was no status, it was as if something was just killing the request mid-way.

I simplified by test script and reduced the number of HTTP calls per iteration, and also performance tuned my SQL queries so they return under 100ms and that helped a great deal. I was able to run the performance test without any mysterious issues and was able to accomplish my goal of figuring out the load our system can handle.

I updated the k6.exe, I tried all the stuff listed here and none of it helped when I was having the issue. But once I did the above, everything just worked as expected. Mysterious indeed.

1 Like