SharedArray usage on large image datasets

Hello all,

Apologies if this was asked and resolved before, I couldn’t find a solution for the same problem.

I have a question about using SharedArray on a scenario with a (relatively) large dataset. As suggested in the documentation, I am opening all the files I need during the init code of the scenario. And this is how I open the files in a loop that runs as long as the file count:

file: open(`../image/documents/${fileName}`, "b"),

This of course opens the file as a binary, and wrapping the original array that includes all the opened files with a SharedArray structure returns the object of type:
map[string]interface {} (I suppose this is a Go type?)

with the following error:

ERRO[0013] invalid type map[string]interface {}, expected string, []byte or ArrayBuffer

I have seen some discussions around the same problem, and tried some of the suggested workarounds like converting the object to an ArrayBuffer type, but that ends up with data inconsistency.

Another problem is that the init code calls the method that opens the files on each VU iteration, rather than once. Which, as expected, creates a huge memory usage and makes the process exhaust all the memory limits, hence failing the run.

So my questions are:

  • Is there currently a way to open unstructured datasets using SharedArray, hence allowing k6 to use the same files, and handle garbage collection properly?
  • If not, is there another way to open all the files just once, outside of the VU code, but allowing each VU to access the same exact array of opened files apart from SharedArrays?

Thanks in advance for any help!

Depends on what exactly exactly you mean by “unstructured”? But probably not yet, watch Shared-memory read-only files · Issue #1931 · grafana/k6 · GitHub for updates on the topic.

Can you please elaborate a bit more on your specific use case and how you are currently trying to use SharedArray? How does your big data file look like and why are you trying to open it as a binary file? And can you paste a fuller snippet of your JS code that is trying to use SharedArray, since from the errors you are getting, I think something isn’t quite right there?