Feature Request: Train using a text iterator

[Santosh-Gupta]

At the moment, it looks like training can only occur using direct paths to text files. This would be tricky if we want to do some custom pre-processing, or train on text contained over a dataset.

A way to train over an iterator would allow for training in these scenarios. An example of this is the sentencepiece library, which allows for an iterator to be used

https://github.com/google/sentencepiece/tree/master/python#training-without-local-filesystem

[n1t0]

Related to #198

[Narsil]

Closing in favor of #198 to keep the discussion centralised (PR open at #512 )