Papermill: Some weird error messages when executing a notebook involving pytorch

[hsl89]

I have a notebook for training a model using pytorch. The notebook runs fine if I run it from browser. But I ran into the following problem when executing it via papermill

Generating grammar tables from /home/ubuntu/anaconda3/envs/pytorch_p36/lib/python3.6/site-packages/blib2to3/Grammar.txt
Writing grammar tables to /home/ubuntu/.cache/black/20.8b1/Grammar3.6.10.final.0.pickle
Writing failed: [Errno 2] No such file or directory: '/home/ubuntu/.cache/black/20.8b1/tmp0twtlmvs'
Generating grammar tables from /home/ubuntu/anaconda3/envs/pytorch_p36/lib/python3.6/site-packages/blib2to3/PatternGrammar.txt
Writing grammar tables to /home/ubuntu/.cache/black/20.8b1/PatternGrammar3.6.10.final.0.pickle
Writing failed: [Errno 2] No such file or directory: '/home/ubuntu/.cache/black/20.8b1/tmp2_jvdud_'
Executing:   0%|                                                                                   | 0/23 [00:00<?, ?cell/s]Executing notebook with kernel: pytorch_p36
Executing:  22%|████████████████▎                                                          | 5/23 [00:03<00:13,  1.38cell/s]
Traceback (most recent call last):
  File "/home/ubuntu/anaconda3/envs/pytorch_p36/bin/papermill", line 8, in <module>
    sys.exit(papermill())
  File "/home/ubuntu/anaconda3/envs/pytorch_p36/lib/python3.6/site-packages/click/core.py", line 764, in __call__
    return self.main(*args, **kwargs)
  File "/home/ubuntu/anaconda3/envs/pytorch_p36/lib/python3.6/site-packages/click/core.py", line 717, in main
    rv = self.invoke(ctx)
  File "/home/ubuntu/anaconda3/envs/pytorch_p36/lib/python3.6/site-packages/click/core.py", line 956, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/home/ubuntu/anaconda3/envs/pytorch_p36/lib/python3.6/site-packages/click/core.py", line 555, in invoke
    return callback(*args, **kwargs)
  File "/home/ubuntu/anaconda3/envs/pytorch_p36/lib/python3.6/site-packages/click/decorators.py", line 17, in new_func
    return f(get_current_context(), *args, **kwargs)
  File "/home/ubuntu/anaconda3/envs/pytorch_p36/lib/python3.6/site-packages/papermill/cli.py", line 256, in papermill
    execution_timeout=execution_timeout,
  File "/home/ubuntu/anaconda3/envs/pytorch_p36/lib/python3.6/site-packages/papermill/execute.py", line 118, in execute_notebook
    raise_for_execution_errors(nb, output_path)
  File "/home/ubuntu/anaconda3/envs/pytorch_p36/lib/python3.6/site-packages/papermill/execute.py", line 230, in raise_for_execution_errors
    raise error
papermill.exceptions.PapermillExecutionError: 
---------------------------------------------------------------------------
Exception encountered at "In [2]":
---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
<ipython-input-2-3916aaf64ab2> in <module>
----> 1 from torchvision import datasets, transforms
      2 
      3 datasets.MNIST('data', download=True, transform=transforms.Compose([
      4     transforms.ToTensor(),
      5     transforms.Normalize((0.1307,), (0.3081,))

~/anaconda3/envs/pytorch_p36/lib/python3.6/site-packages/torchvision/__init__.py in <module>
      1 import warnings
      2 
----> 3 from torchvision import models
      4 from torchvision import datasets
      5 from torchvision import ops

~/anaconda3/envs/pytorch_p36/lib/python3.6/site-packages/torchvision/models/__init__.py in <module>
     10 from .shufflenetv2 import *
     11 from . import segmentation
---> 12 from . import detection
     13 from . import video
     14 from . import quantization

~/anaconda3/envs/pytorch_p36/lib/python3.6/site-packages/torchvision/models/detection/__init__.py in <module>
----> 1 from .faster_rcnn import *
      2 from .mask_rcnn import *
      3 from .keypoint_rcnn import *

~/anaconda3/envs/pytorch_p36/lib/python3.6/site-packages/torchvision/models/detection/faster_rcnn.py in <module>
     11 
     12 from .generalized_rcnn import GeneralizedRCNN
---> 13 from .rpn import AnchorGenerator, RPNHead, RegionProposalNetwork
     14 from .roi_heads import RoIHeads
     15 from .transform import GeneralizedRCNNTransform

~/anaconda3/envs/pytorch_p36/lib/python3.6/site-packages/torchvision/models/detection/rpn.py in <module>
      9 from torchvision.ops import boxes as box_ops
     10 
---> 11 from . import _utils as det_utils
     12 from .image_list import ImageList
     13 

~/anaconda3/envs/pytorch_p36/lib/python3.6/site-packages/torchvision/models/detection/_utils.py in <module>
     17 
     18 @torch.jit.script
---> 19 class BalancedPositiveNegativeSampler(object):
     20     """
     21     This class samples batches, ensuring that they contain a fixed proportion of positives

~/anaconda3/envs/pytorch_p36/lib/python3.6/site-packages/torch/jit/__init__.py in script(obj, optimize, _frames_up, _rcb)
   1217         if _rcb is None:
   1218             _rcb = _jit_internal.createResolutionCallback(_frames_up + 1)
-> 1219         _compile_and_register_class(obj, _rcb, qualified_name)
   1220         return obj
   1221     else:

~/anaconda3/envs/pytorch_p36/lib/python3.6/site-packages/torch/jit/__init__.py in _compile_and_register_class(obj, rcb, qualified_name)
   1074 def _compile_and_register_class(obj, rcb, qualified_name):
   1075     ast = get_jit_class_def(obj, obj.__name__)
-> 1076     _jit_script_class_compile(qualified_name, ast, rcb)
   1077     _add_script_class(obj, qualified_name)
   1078 

~/anaconda3/envs/pytorch_p36/lib/python3.6/site-packages/torch/jit/_recursive.py in try_compile_fn(fn, loc)
    220     # object
    221     rcb = _jit_internal.createResolutionCallbackFromClosure(fn)
--> 222     return torch.jit.script(fn, _rcb=rcb)
    223 
    224 

~/anaconda3/envs/pytorch_p36/lib/python3.6/site-packages/torch/jit/__init__.py in script(obj, optimize, _frames_up, _rcb)
   1224         if _rcb is None:
   1225             _rcb = _gen_rcb(obj, _frames_up)
-> 1226         fn = torch._C._jit_script_compile(qualified_name, ast, _rcb, get_default_args(obj))
   1227         # Forward docstrings
   1228         fn.__doc__ = obj.__doc__

RuntimeError: 
builtin cannot be used as a value:
at /home/ubuntu/anaconda3/envs/pytorch_p36/lib/python3.6/site-packages/torchvision/models/detection/_utils.py:14:56
def zeros_like(tensor, dtype):
    # type: (Tensor, int) -> Tensor
    return torch.zeros_like(tensor, dtype=dtype, layout=tensor.layout,
                                                        ~~~~~~~~~~~~~ <--- HERE
                            device=tensor.device, pin_memory=tensor.is_pinned())
'zeros_like' is being compiled since it was called from '__torch__.torchvision.models.detection._utils.BalancedPositiveNegativeSampler.__call__'
at /home/ubuntu/anaconda3/envs/pytorch_p36/lib/python3.6/site-packages/torchvision/models/detection/_utils.py:72:12

            # randomly select positive and negative examples
            perm1 = torch.randperm(positive.numel(), device=positive.device)[:num_pos]
            perm2 = torch.randperm(negative.numel(), device=negative.device)[:num_neg]

            pos_idx_per_image = positive[perm1]
            neg_idx_per_image = negative[perm2]

            # create binary mask from indices
            pos_idx_per_image_mask = zeros_like(
            ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~...  <--- HERE
                matched_idxs_per_image, dtype=torch.uint8
            )
            neg_idx_per_image_mask = zeros_like(
                matched_idxs_per_image, dtype=torch.uint8
            )

            pos_idx_per_image_mask[pos_idx_per_image] = torch.tensor(1, dtype=torch.uint8)
            neg_idx_per_image_mask[neg_idx_per_image] = torch.tensor(1, dtype=torch.uint8)

[willingc]

Hi @hsl89, Are you running papermill in a conda environment? Also, please double check that you are using the latest papermill release. Thanks :sunny:

[hsl89]

hey @willingc, that's exactly right. I am running in a conda environemnt. I did have the latest version of papermill (2.3.1)

[MSeal]

Does the call work without papermill? This looks like an environment issue below papermill on executing the code snippet independent of Notebook.

[hsl89]

hey @MSeal , the notebook does run if I execute it manually from the browser. I have done such "A/B testing" before posting the issue. To help you troubleshoot, I am trying run this guy:
https://github.com/aws/amazon-sagemaker-examples/blob/master/hyperparameter_tuning/pytorch_mnist/hpo_pytorch_mnist.ipynb

[MSeal]

Is your notebook execution using the same ~/anaconda3/envs/pytorch_p36 conda environment?

[hsl89]

Yes

[hsl89]

@MSeal , please let me know if you can reproduce the problem. happy to provide more input

[MSeal]

For transparency I am not going to have time to dig deeply into reproducing. New baby and day job means I only have a few minutes each work day to look at the issues across a dozen projects. I would try narrowing down the scope of the problem and see if you can reproduce it in fresh environments. Also look up the compiled error message on pytorch. They are more likely to know what's wrong here as well

[hsl89]

Thanks for being transparent @MSeal . let's do this, I will close this issue now. and I will run some tests on notebooks that only involves import statements in different envs with different versions of pytorch. and I will work with my team to narrow down the cause. To match up your transparency, we do have plan to use papermill on one of our important projects at SageMaker. It would be great if you can loop in another folk (who doesn't have to work on the new born baby problem) to help us resolving this issue.

[MSeal]

@hsl89 Btw there is a weekly nteract meeting https://github.com/nteract/meeting-minutes where we all discuss topics overall in the oss org. If there's a higher initiative and interest in using tooling in specific or new ways it can be helpful to discuss with a wider group there where we've dedicated time. We usually plan topics ahead of time in the #weekly-meeting channel of our slack https://nteract.slack.com/. Overall it's mostly me and Carol maintaining papermill atm with many contributors for individual PRs. We are welcoming of having more people involved in greater capacity :D

[hsl89]

Hi @MSeal , thanks for letting me know about it. I am interested to learn more about the papermill community. I tried to login to the slack workspace with my email, but it says my email does not have an account there. So I guess you will have to invite me ([email protected])

[willingc]

@hsl89 Invite sent. Happy to have you join us.