Pickle dumps not readable between python 2/3

[kain88-de]

The problem is that we always write using the highest available protocol. But I was unaware that python3 introduced a new protocol version

https://stackoverflow.com/questions/25843698/valueerror-unsupported-pickle-protocol-3-python2-pickle-can-not-load-the-file#25843743

To have pickles load across python version and be reasonable fast/small we have to explicitly choose version 2 of the protocol.

[richardjgowers]

What are we using pickles for?

[kain88-de]

python data pickles. For example if we store python objects.

[richardjgowers]

Right yeah, but what is datreant using pickle for? git grep pickle isn't showing me anything?

[kain88-de]

datreant.data/src/datreant/data/pydata.py

Line 41 in 89f8937

pickle.dump(data, self.handle, pickle.HIGHEST_PROTOCOL)

Sure you are on the right repository?

[richardjgowers]

Ah right yeah, I was looking at core

[kain88-de]

Oh well with string handling this is even more broken -.-. We have to check reading/writing with the bytes flags rb/wb to ensure data can be read/written independent of the python version.

[kain88-de]

This is to say. I now also encountered that a python2 dump cannot be read by python 3

[kain88-de]

https://bugs.python.org/issue6784

So they claim this is resolved. But I honestly haven't found a good solution today to enable reading / writing of pickles between python versions that works problem free. Some simple stiff and builtin types might work but not complex objects.