csv upload fails with 117k respondents

[manumoreira]

CDC tested the CSV upload with a real sample file of 117k and even if the file was finally uploaded there were some issues:
In some cases, this error shows-up

In some others, the uploading process last forever in the respondents list drag and drop box

Here's an example
117k.csv

[ysbaddaden]

Some context from @matiasgarciaisaia: the production server only allocated 512MB of memory to Surveda, which crashed while importing the CSV (out-of-memory). Allowing Surveda to use 1GB of memory avoids the problem.

As suggested before, we shall be able to reduce the memory usage by streaming the respondents in batches from the controller to the actual function, allowing to work with batches of 1000 respondents at once instead of all respondents. It would still take time to process all the respondents, but memory usage would stay low, and would likely help to only have to deal with 1000 respondents at most.

[manumoreira]

Adding a reference to the latest tests on this https://manas.slack.com/archives/C1L7CJMNU/p1683736128357109