Unfortunately, at the moment, when you open a file in k6, each VU has its own individual copy of that file in memory. There’s also no way at the moment to not read a whole file in memory at once, but read it line by line, in a streaming manner. Both of these things are high in our roadmap: https://github.com/loadimpact/k6/issues/532, https://github.com/loadimpact/k6/issues/592, https://github.com/loadimpact/k6/issues/1021#issuecomment-493155860
What type of a data store are you using? I’m asking because we’ve noticed that CSV parsing with some popular JS libraries like papaparse takes up a surprisingly large amount of RAM, so if that’s the case with you, directly loading JSON files or plain text files might be a partial short-term workaround.
There are other tricks you can use to reduce k6 memory usage (like
discardResponseBodies and the upcoming
--compatibility-mode=base option), but these won’t fully make up for a huge static file loaded in each VU. Unfortunately, until we fix the underlying issues, we’re unlikely to support millions of datastore records with lots of VUs on the same machine So until then, you’d need either a bigger machine and/or smaller datastore files and/or less VUs per machine…