Aggregate fails when output_single_file_only option set to False

[gwaybio]

In an experiment with >1,000 sites, the aggregate recipe step fails quietly. We do not observe any errors, but the recipe next step is nevertheless performed and not surprisingly fails.

This may be a compute size issue, but it silently failing is still concerning and we should address.

One option is to aggregate each site independently, and then, with the number of single cells per perturbation, weight the aggregated contribution proportionally to cell count. I describe this option in #57 - time to revisit!

[gwaybio]

One option is to aggregate each site independently, and then, with the number of single cells per perturbation, weight the aggregated contribution proportionally to cell count.

This is only an approximation to the aggregation and cannot be exactly equivalent to aggregating all single cells at once.

In #70 I implement an approx_aggregate_piecewise() function that does precisely the above. Currently, I have it set to only approx aggregate on sites, but we might explore aggregating based on wells as well.