Torch7: Back-compatibility deserialisation issue

[Atcold]

Here there is a model I've trained some time ago.

https://engineering.purdue.edu/elab/smartEYE-data/net/17cate9filter.tar.gz

With the latest version of torch / cunn the object is no longer deserialisable.
This is awful. I can no longer access my previously trained models.

Reverting back torch / cunn (and many more packages) to, say, 1 year ago, fixes the problem.

th> require 'cunn';
                                                                      [0.9739s] 
th> net = torch.load('model-127.net')
/home/atcold/torch/install/share/lua/5.1/torch/File.lua:301: Failed to load function from
 bytecode: (binary): cannot load incompatible bytecode
stack traceback:
    [C]: in function 'error'
    /home/atcold/torch/install/share/lua/5.1/torch/File.lua:301: in function 'readObject'
    /home/atcold/torch/install/share/lua/5.1/torch/File.lua:369: in function 'readObject'
    /home/atcold/torch/install/share/lua/5.1/nn/Module.lua:154: in function 'read'
    /home/atcold/torch/install/share/lua/5.1/torch/File.lua:351: in function 'readObject'
    /home/atcold/torch/install/share/lua/5.1/torch/File.lua:369: in function 'readObject'
    /home/atcold/torch/install/share/lua/5.1/torch/File.lua:369: in function 'readObject'
    /home/atcold/torch/install/share/lua/5.1/nn/Module.lua:154: in function 'read'
    /home/atcold/torch/install/share/lua/5.1/torch/File.lua:351: in function 'readObject'
    /home/atcold/torch/install/share/lua/5.1/torch/File.lua:369: in function 'readObject'
    /home/atcold/torch/install/share/lua/5.1/torch/File.lua:369: in function 'readObject'
    /home/atcold/torch/install/share/lua/5.1/nn/Module.lua:154: in function 'read'
    /home/atcold/torch/install/share/lua/5.1/torch/File.lua:351: in function 'readObject'
    /home/atcold/torch/install/share/lua/5.1/torch/File.lua:409: in function 'load'
    [string "net = torch.load('model-127.net')"]:1: in main chunk
    [C]: in function 'xpcall'
    /home/atcold/torch/install/share/lua/5.1/trepl/init.lua:669: in function 'repl'
    ...cold/torch/install/lib/luarocks/rocks/trepl/scm-1/bin/th:199: in main chunk
    [C]: at 0x00406670  
                                                                      [0.1411s] 

[soumith]

This happens when:

1. your model has serialized functions
   AND
2. you changed lua versions (for example went from LuaJIT 2.0 to LuaJIT 2.1, or Lua 5.2 for example)

This is because when we serialize functions, we use Lua's inbuilt function serialization support.
Usually none of our models have serialized functions, but yours seems to have one.

I would suggest loading the model with whatever worked, loop over the modules and nil all the functions, and saving it again. hope that helps.

[Atcold]

OMG.

Thank you, @soumith. I need to kill an ex CS student of ours.

BTW, 1. == true but 2. == false! I'm still on the same LuaJIT. I've simply reverted Torch back one year from today. Or, could you tell me how I can verify which Lua (JIT 2.0/.1 5.1/2) is used?

[soumith]

if 2 == false, i will investigate your provided test case further. I'll have to git bisect File.lua and isolate the problematic commit. What is the last known good commit that got things working for you? If you have time and can do a git bisect, that will save me quite some time as well (but if you dont, i understand).

[Atcold]

18 Nov 2015 I was working with this net, and I assume I was on a freshly updated system.

[apaszke]

@soumith torch/nn#749 can reduce the number of such issues :grin:

[Atcold]

@soumith, do we have any progress on this issue?
By removing the overwritten method (model.modules[1].updateGradInput = function(...) end) I am now able to load the model with the most updated Torch. Still, I think this is simply a workaround.

[soumith]

If you do a git bisect and give me the exact commit hash that changed it from working to non-working, that'll really help me. it wont take you long to do, as it's a binary search.

[Atcold]

OK, I'm going to have a look at https://git-scm.com/book/en/v2/Git-Tools-Debugging-with-Git#Binary-Search

Since Torch is multi-packaged, how do you suggest to move across history?
Shall I keep synchronising torch7, nn, cutorch and cunn over every iteration, correct?

[soumith]

just do a bisect on https://github.com/torch/distro

[Atcold]

Breaking commit: 6fa0c170b4eac43360d48f2770971353b9ed2420.
Still good one (one commit before): 5c9b74acad5d69047943906811828734191c66f2.
Sorry for the delay.

[Atcold]

The only commit touching File.lua, in torch package, is 99a914364018278b37aa1fd6f5692237acf32a8d which is simply issuing an error if loadstring() returns a non-OK status.
Alright, it looks like the torch package is not responsible. Trying something else now.

[soumith]

So, your models weren't loading fully before that commit either -- it was simply that the commit errors out functions that could not load (previous to that, i guess it would silently not complain)

[Atcold]

No, no, it was not that. Reverting only torch doesn't help. It still breaks.
I'm double checking right now.

[Atcold]

So, indeed the line

error(string.format('Failed to load function from bytecode: %s', err))

Introduced the breakage. Now I'm wondering whether it makes sense to go back, bisect once more, while adding that line to File.lua, so that we see where actually things got corrupted. What do you think? Is there any other smarter way?

[soumith]

there's no smarter way. I could introduce a flag in torch to make this error to be a warning. That will atleast load the rest of your model, ignoring the function loading failures...

[nicholas-leonard]

I have also seen this error pop-up with older models.

[Atcold]

So, I went back all the way to the origin of this repository, checking manually for and error in loadstring(dumped) and, yes, it's been there since the beginning.
I am now thinking that this could have been introduced when we moved from _Lua_ to _LuaJIT_, perhaps.
So, I wouldn't know, now, what else to try. Well, one could try to install torch with a different backend, but I don't know how to do so. I'm open to suggestions, although I think it's becoming a little too much work for something I'm no longer using. Nevertheless, fixing back-compatibility issues is important, I think.

[aparnamhnty]

I am new to Torch.....I want to implement the code given at http://torch.ch/blog/2015/07/30/cifar.html
While trying to execute train.lua........it initially shows the parameters and configures the model, but while loading the data, it complains that 'Failed to load function from bytecode: (binary): cannot load incompatible bytecode.
What shall I do to solve this problem.
'

[ArturoDeza]

Bump. I had a similar issue and fixed it by switching between CUDA versions (9.2 and 8.0). Note: installing another CUDA version side-by-side was not the end of the world (as I thought it would be), as it is fine to have multiple versions installed as long as the PATH and LD_LIBRARY_PATH variables are appropriately synced for the application you are using.
I also noticed that for some reason when doing work in style transfer, rendering quality changes drastically depending on use of LUA or LUAJIT for the torch compiler. I will to submit a report once (or if), I can track the bug .