Feasibility study: ditch farm and use pytorch

[Shreyanand]

Estimate the benefit, risk, and labor cost involved in removing the farm architecture and using simple PyTorch. We want to see if we do that, then does our workflow that involves quay image and Kubeflow pipelines become easy to understand, simplified, and less error-prone.

[MichaelTiemannOSC]

Noting that FARM project has transitioned to HAYSTACK: https://github.com/deepset-ai/haystack/ and that some OS-Climate members are also HAYSTACK sponsors/supporters.

[Shreyanand]

There are two alternatives that seems plausible here:

1. Using a solution like mentioned here

   Benefits:

   • retriever and reader are in the same pipeline so simpler architecture
   • latest models using the HAYSTACK library
   • may solve the multiprocessing issue with kpi-train

   Risk and Cost:

   • may introduce new errors while running in the container env.
   • 1-2 week developer time to convert data and results into the right format

2. Using huggingface QA (transformer package)

   Benefits:

   • the root library used as a standard for transformers so it gives a lot of flexibility
   • may solve the multiprocessing issue with kpi-train

   Risk and Cost:

   • may introduce new errors while running in the container env.
   • 1-2 week developer time to convert data and results into the right format

@erikerlandson

[MichaelTiemannOSC]

Let's not forget that we have a technical solution (disable multiprocessing) and a view to a technical solution (allocating appropriate amounts of SHM) to the train kpi-extraction problem. If there are OTHER problems with farm we want to solve, let's talk about those, but we should be able to dispose of the kpi-extaction problem very simply.

[Shreyanand]

The disable multiprocessing solution makes the training almost impractical, in one experiment it took around 1.5 days to train 80% only to run into kernel error. This was for training with 145 files. When we get more annotated files, it would be even more difficult. Allocating appropriate amounts of SHM seems like longer term solution to get working.

One other obvious problem with FARM is, like @MichaelTiemannOSC pointed out, that it is no longer actively maintained. It has transitioned to farm-haystack and any further development will happen in that pypi package. In my above comment, I list two alternatives, one is using farm-haystack that is actively maintained or using transformers package which is the root package used by farm-haystack to build upon.

[erikerlandson]

We should absolutely get off of the unmaintained package. We could use farm-haystack but it's an added layer of dependency and I'm trying to sort out the cost/benefits over just using transformers package.

cc @JeremyGohBNP @ChristianMeyndt @idemir-ids @andraNew