Papermill: using with minio

[thbeh]

Hi, how would I use papermill with notebooks in minio? is there a way to pass in endpoint-url?

[thbeh]

I got this trying to run papermill with jupyter in S3 (minio)

root@1b835ae0f3d0:~# papermill s3://s3contents-demo/notebooks/nb1.ipynb s3://s3contents-demo/notebooks/nb2.ipynb -p message 2
Input Notebook: s3://s3contents-demo/notebooks/nb1.ipynb
Output Notebook: s3://s3contents-demo/notebooks/nb2.ipynb
Found credentials in shared credentials file: ~/.aws/credentials
Traceback (most recent call last):
File "/usr/local/bin/papermill", line 10, in
sys.exit(papermill())
File "/usr/local/lib/python3.6/dist-packages/click/core.py", line 764, in __call__
return self.main(args, kwargs)
File "/usr/local/lib/python3.6/dist-packages/click/core.py", line 717, in main
rv = self.invoke(ctx)
File "/usr/local/lib/python3.6/dist-packages/click/core.py", line 956, in invoke
return ctx.invoke(self.callback, *ctx.params)
File "/usr/local/lib/python3.6/dist-packages/click/core.py", line 555, in invoke
return callback(args, *kwargs)
File "/usr/local/lib/python3.6/dist-packages/papermill/cli.py", line 165, in papermill
cwd=cwd,
File "/usr/local/lib/python3.6/dist-packages/papermill/execute.py", line 74, in execute_notebook
nb = load_notebook_node(input_path)
File "/usr/local/lib/python3.6/dist-packages/papermill/iorw.py", line 369, in load_notebook_node
nb = nbformat.reads(papermill_io.read(notebook_path), as_version=4)
File "/usr/local/lib/python3.6/dist-packages/papermill/iorw.py", line 93, in read
notebook_metadata = self.get_handler(path).read(path)
File "/usr/local/lib/python3.6/dist-packages/papermill/iorw.py", line 208, in read
return "\n".join(S3().read(path))
File "/usr/local/lib/python3.6/dist-packages/papermill/s3.py", line 429, in read
for block in self.cat(source, compressed=compressed, encoding=encoding):
File "/usr/local/lib/python3.6/dist-packages/papermill/s3.py", line 297, in cat
size = obj.content_length
File "/usr/local/lib/python3.6/dist-packages/boto3/resources/factory.py", line 339, in property_loader
self.load()
File "/usr/local/lib/python3.6/dist-packages/boto3/resources/factory.py", line 505, in do_action
response = action(self, args, *kwargs)
File "/usr/local/lib/python3.6/dist-packages/boto3/resources/action.py", line 83, in __call__
response = getattr(parent.meta.client, operation_name)(params)
File "/usr/local/lib/python3.6/dist-packages/botocore/client.py", line 357, in _api_call
return self._make_api_call(operation_name, kwargs)
File "/usr/local/lib/python3.6/dist-packages/botocore/client.py", line 661, in _make_api_call
raise error_class(parsed_response, operation_name)
botocore.exceptions.ClientError: An error occurred (403) when calling the HeadObject operation: Forbidden

[MSeal]

So the error message you got was because the credentials on the machine running the papermill command (it printed it found ~/.aws/credentials) did not have permissions to read the input notebook path. I'm not super familiar with minio setups, but if you can use s3 commands it should also work for papermill.

[thbeh]

Thanks. S3 commands work well with endpoint-url plugin installed.

[thbeh]

Using the python api, would I be able to overide the s3 session, adding an additional endpoint-url parameter? In boto3 I would do s3 = boto3.resource('s3', ‘endpoint_url=http://minio:9000’)

[xmnlab]

I was trying to figure out the best way to resolve this problem.

• the issue related this problem in boto3 was closed, not resolved (https://github.com/boto/boto3/issues/1375). The recommendation there was set this during client creation as an argument.
• this discussion on aws-cli is still opened (https://github.com/aws/aws-cli/issues/1270).

so I am proposing in #415 the usage of an environment variable called BOTO3_ENDPOINT_URL

any feedback on PR #415 is very welcome!

[thbeh]

As I understand S3 sessions will pick credentials from .aws/credentials file created. There is also a config file .aws/config that provides additional configuration for custom profiles. Using https://github.com/wbingli/awscli-plugin-endpoint, a config file like below is created. Using normal aws s3 ls, I could see all files in my S3. My 2 cents
[default]
region = us-east-1
s3 =
signature_version = s3v4
endpoint_url = http://minio:9000
[plugins]
endpoint = awscli_plugin_endpoint

[MSeal]

Yes the preferred pattern is probably a config file. If someone works through getting aws/config functioning for papermill's setup I'd gladly merge that as well.

[motin]

As I understand S3 sessions will pick credentials from .aws/credentials file created.

I think it goes something like this: First it will use arguments supplied when creating the session, then look for env vars, then look for .aws/credentials. So if you supply env vars you should not need the .aws/credentials file.

Maybe https://github.com/mozilla/OpenWPM/blob/1d4254eb673342146340511427b50a7fdf7b1f46/test/utilities.py#L142-L185 is relevant - it is for localstack but the principle is the same.

[ca-scribner]

This works for me:

export BOTO3_ENDPOINT_URL=http://my.minio:9000
export AWS_ACCESS_KEY_ID=my_minio_key
export AWS_SECRET_ACCESS_KEY=my_minio_secret
papermill s3://my-bucket/inputNotebook.ipynb s3://my-bucket/outputNotebook.ipynb