Fairseq: BacktranslationDataset doesn't work with multiple workers

Created on 13 Jun 2019  路  2Comments  路  Source: pytorch/fairseq

if num_workers > 0, there will be an error:
AttributeError: Can't pickle local object 'SemisupervisedTranslationTask.build_model.<locals>.backtranslate_fn'

enhancement help wanted
Source
picture hxbai

Most helpful comment

Thanks for reporting! I will need to find some workaround of it. For now we can only use one worker if we want to use BacktranslationDataset

picture pipibjc on 14 Jun 2019
👍3

All 2 comments

Yeah, I'm not sure BacktranslationDataset was ever intended to be run with multiple workers since it relies on models that are on the GPU. cc @pipibjc

lematt1991 picture lematt1991 on 14 Jun 2019

Thanks for reporting! I will need to find some workaround of it. For now we can only use one worker if we want to use BacktranslationDataset

pipibjc picture pipibjc on 14 Jun 2019
👍3
Was this page helpful?
0 / 5 - 0 ratings

Related issues

jordiae picture jordiae  路  3Comments
galphag picture galphag  路  3Comments
kyquang97 picture kyquang97  路  3Comments
tyoc213 picture tyoc213  路  3Comments
zqs01 picture zqs01  路  3Comments