Hi @rodyb,
I would recommend reading Unique file upload per iteration - #3 by mstoykov while not the same as your case the current general advice on “how to upload a lot of different files” is somewhere between:
- don’t
- try to generate them when needed instead of loading them
- figure some other way to load test that doesn’t involve it
- combine multiple hacks to make it more viable/possible
Arguably you can figure out some way that works for your, but in general:
- you need to load files in the init context with pure k6. You can use xk6 and build an extension to circumvent this - but that likely won’t help you much - see 3.
- Unless you only load some files in some VUs - essentially sharding the files across VUS. you will need at least as much memory as * in reality it will be that times 2-3x due to inefficiencies.
- As explained in detail in this issue the problem are many and can’t really be fixed (IMO) with simple patches to the
k6/httpAPI. Again depending on your needs and desire to work around it you can probably write a lot of this as an xk6 extension. Although that one will need to emithttp_req*metrics in order to be at least somewhat useful IMO.
Depending on all of your circumstances … you might just be okay with loading 10 files that go up to 200mb together and load testing with 20 VUs. Or maybe you can load test with 200 VUs but sharding the files in some tricky way between them so that it works for you and still not need hundreds of GBs of memory.
The exact specifics will depend solely on your exact situation.
Hope this helps you, and anyone who has done something similar is welcome to give their input