Ml-agents: Ctrl-C not saving/exporting at end of training

[NightSp4rk]

As the title says.
After some digging, I saw that KeyboardInterrupt exception was not being called, instead giving the following error:

forrtl: error (200): program aborting due to control-C event
Image PC Routine Line Source

libifcoremd.dll 00007FFB5F4194C4 Unknown Unknown Unknown
KERNELBASE.dll 00007FFB982DEC3F Unknown Unknown Unknown
KERNEL32.DLL 00007FFB9AE113D2 Unknown Unknown Unknown
ntdll.dll 00007FFB9B0C54E4 Unknown Unknown Unknown

I have altered the trainer_controller.py to save and export every few thousand steps, as a way to bypass this issue. But I'm wondering if it's just me, and why am I facing this issue.
Thanks

[eshvk]

This seems related to this issue. I am assuming you are on Windows and you have loaded the latest master.

[NightSp4rk]

I am indeed using the latest from master. I think this may be a different issue as the error I got is different from the one at that link, and since I am getting this when it should already have been fixed.

[KReguieg]

HINT IF AFRAID OF LOOSING MODEL AFTER HOURS OF TRAINING BY PRESSING CTRL-C

We had the same issue, and thought our model is lost, but refering to @Hengoo answer you can load the model and continue training at a given point in time, and then exit the training in Unity by pressing the Play-Button, so your model get's saved. Worked for us and we could save our model of 20+ hours of training! ;)

[xiaomaogy]

This bug has been fixed with https://github.com/Unity-Technologies/ml-agents/pull/1558. The error message

forrtl: error (200): program aborting due to control-C event
Image PC Routine Line Source
libifcoremd.dll 00007FFDAFF994C4 Unknown Unknown Unknown
KERNELBASE.dll 00007FFDEF127EDD Unknown Unknown Unknown
KERNEL32.DLL 00007FFDF2131FE4 Unknown Unknown Unknown
ntdll.dll 00007FFDF226EFC1 Unknown Unknown Unknown

still exist, which is related to this stackoverflow https://stackoverflow.com/questions/42653389/ctrl-c-causes-forrtl-error-200-rather-than-python-keyboardinterrupt-exception. Our fix get around this error by adding another ctrl-c event handler which saves the model before this error.

[lock[bot]]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.