Gensim: Loading models generated by other version of gensim

[k0nserv]

Hello 👋,

I just tried loading a word2vec model generated with 0.13.4.1 in 0.13.1 which failed with errors relating to pickle. Specifically errors like this:

 E           AttributeError: Can't get attribute 'call_on_class_only' on <module 'gensim.utils' from '/usr/local/lib/python3.5/site-packages/gensim/utils.py'>

This confused me a lot initialy, but it seems that this is happening due to the use of pickle to serialize/deserialize Python object. call_on_class_only was not available in 0.13.1 so when trying to deserialize a file saved with a newer version of the library. However given that the different versions of the library were similar I assumed that loading and saving would work across any version in the 0.x.y space. With semver in mind a 0.x.y version does indicate that the project is not yet stable so I guess this is to be expected, all though semver only mentions that the public interface should be considered unstable(not artifcats such as save files).

When going forward with 1.x.y As a user I'd expect that any file saves with a 1.x.y version should be backwards compatible with any other 1.x.y version.

Thanks

[gojomo]

Looks like an unintended side-effect of #889. Here it may be fixable with some special tear-down/fix-up in load()/save(), to avoid unnecessarily complicating the on-disk format.

As a general matter, though, I don't think we'd want to target older-version load()-foward-compatibility for newer-version saves, within MAJOR.x.y version families for varying x/y.

Many fixes/improvements become harder if needing to consider load-compatibility with a large number of prior versions. The other direction – continuing to be able to use older models and code unchanged after upgrades – is far more important. And if that direction is done really well, that also provides the best solution for people wanting to use newer-version models elsewhere – upgrade elsewhere, because it's generally safe to do so.

[k0nserv]

As a general matter, though, I don't think we'd want to target older-version load()-foward-compatibility for newer-version saves, within MAJOR.x.y version families for varying x/y.

Fair enough, maybe a better error message could be provided though? I had to spend quite some time reading the source and errors to figure out what was going wrong.

Many fixes/improvements become harder if needing to consider load-compatibility with a large number of prior versions. The other direction – continuing to be able to use older models and code unchanged after upgrades – is far more important. And if that direction is done really well, that also provides the best solution for people wanting to use newer-version models elsewhere – upgrade elsewhere, because it's generally safe to do so.

I know very little about the underlying of Gensim and Pickle, but the use of Pickle seems to make it more fragile than it needs to be. Seems like it might be possible to accidentally break backwards compatibility for save/load by just changing the structure of the object slightly(even in a non breaking fashion)

[menshikh-iv]

Not relevant for now. The actual version is 3.0.0.