Nbconvert: Nbconvert 4.2.0 and output in subdirectory

[pierrepo]

Hi,

nbconvert seems craching when trying to write ouput file in a subdirectory.

In the notebook_test directory I have one simple notebook file test.ipynb. The latter contains a single cell with print("hello world").

When converting to Markdown, nbconvert crashes:

pierre@kiwi> jupyter-nbconvert --to markdown ./notebook_test/test.ipynb --output ./notebook_test/test.md
[NbConvertApp] Converting notebook ./notebook_test/test.ipynb to markdown
[NbConvertApp] Writing 55 bytes to ./notebook_test/./notebook_test/test.md
Traceback (most recent call last):
  File "/home/pierre/.virtualenvs/new_jupyter/bin/jupyter-nbconvert", line 11, in <module>
    sys.exit(main())
  File "/home/pierre/.virtualenvs/new_jupyter/local/lib/python2.7/site-packages/jupyter_core/application.py", line 267, in launch_instance
    return super(JupyterApp, cls).launch_instance(argv=argv, **kwargs)
  File "/home/pierre/.virtualenvs/new_jupyter/local/lib/python2.7/site-packages/traitlets/config/application.py", line 596, in launch_instance
    app.start()
  File "/home/pierre/.virtualenvs/new_jupyter/local/lib/python2.7/site-packages/nbconvert/nbconvertapp.py", line 293, in start
    self.convert_notebooks()
  File "/home/pierre/.virtualenvs/new_jupyter/local/lib/python2.7/site-packages/nbconvert/nbconvertapp.py", line 457, in convert_notebooks
    self.convert_single_notebook(notebook_filename)
  File "/home/pierre/.virtualenvs/new_jupyter/local/lib/python2.7/site-packages/nbconvert/nbconvertapp.py", line 429, in convert_single_notebook
    write_results = self.write_single_notebook(output, resources)
  File "/home/pierre/.virtualenvs/new_jupyter/local/lib/python2.7/site-packages/nbconvert/nbconvertapp.py", line 390, in write_single_notebook
    output, resources, notebook_name=notebook_name)
  File "/home/pierre/.virtualenvs/new_jupyter/local/lib/python2.7/site-packages/nbconvert/writers/files.py", line 127, in write
    with io.open(dest, 'w', encoding='utf-8') as f:
IOError: [Errno 2] No such file or directory: './notebook_test/./notebook_test/test.md'

However, it works perfectly when there is no subdirectory in the output filename:

pierre@kiwi> jupyter-nbconvert --to markdown ./notebook_test/test.ipynb --output test.md
[NbConvertApp] Converting notebook ./notebook_test/test.ipynb to markdown
[NbConvertApp] Writing 55 bytes to ./notebook_test/test.md

The funny thing is that the output file is also written in the same directory of the notebook file.
This does not appear with nbconvert 4.1.0.

I am not sure if this is a bug or a feature.

[timstaley]

I'm also seeing this with 4.2.0 - nbconvert seems to be effectively changing working directory to the directory of the notebook(?) - so then the relative subdirectory path becomes invalid.

As a quick workaround I'm simply passing nbconvert absolute paths.

[herr-biber]

I encounter a similar bug. If you give a relative --output path, its relative to the source ipynb

jupyter nbconvert --to html foo/test.ipynb --output test.html
[NbConvertApp] Converting notebook foo/test.ipynb to html
[NbConvertApp] Writing 264082 bytes tofoo/test.html

By bisecting, I found the bug got introduced by commit dd847573ea2111c775c5dc918a6dbdb76b4b9c2f.
Prior, a relative --output path was relative to $PWD.

Or is this actually a feature?

[minrk]

Arg, no, I think --output should probably be interpreted relative to PWD.

[mpacer]

is this supposed to set build_directory when the --output flag is passed in(using the @observe feature of traitlets)? If so, I think the problem is that its not being caught and the resources directory is always getting passed in and appended to the front of the relative path.

This seems to be happening because even when you specify --output, the change['new'] value is ''.

[mpacer]

So I think that part of this actually isn't a bug.

The flag for an output directory is supposed to be --output-dir not --output.

And build_directory is changed appropriately if you use --output-dir

So change

jupyter-nbconvert --to markdown ./notebook_test/test.ipynb --output ./notebook_test/test.md

to

jupyter-nbconvert --to markdown ./notebook_test/test.ipynb --output-dir ./notebook_test/test.md

or even better

jupyter-nbconvert --to markdown ./notebook_test/test.ipynb --output-dir ./notebook_test --output test.md

but note that if you try

jupyter-nbconvert --to markdown ./notebook_test/test.ipynb --output-dir ./notebook_test/test2.md --output test.md 

the output will be to test2.md not test.md.

[herr-biber]

I don't get why there are two output parameters? Why can't we have only a single output parameter (e.g. --output) and rely on shell semantics for relative vs. absolute path handling?

[takluyver]

You can convert several notebooks with one command to a specified --output-dir. --output only makes sense when converting one.

[mpacer]

Ah — so should --output not be behaving in this way? Is it actually a bug?

[takluyver]

I agree with what Min said above that output should probably be relative to the CWD where you run the command, not the location of the notebook. Is that what you mean?

[mpacer]

Not quite, though that was also something I was trying to work out. It looks like there are like three potential issues here.

What I was asking was whether --output is supposed to take paths or only file names (including relative paths).

--output is not triggering build-directory, but it doesn't seem to have been designed to do so. If it had the first reported error on this issue.

--output is not triggering relpath either, even when the path passed in is relative. It is not clear what relpath is used for or where it would be triggered.

I'm not sure where else the code determining whether there was a relative or an absolute path would be… --output doesn't seem to be processed in the FilesWriter.

It's not clear how the cited changes are able to mess with which directory the relative path is being defined from. It seems that the resources path is being used in all of the cases and that relpath is just never defined. But I wasn't certain, I've been trying to piece this together as best I can.

[mpacer]

This seems to be related to problems reported in #326.

[mpacer]

Ok, so --output has a particular meaning according to its associated help on the output_base traitlet:

overwrite base name use for output files. can only be used when converting one notebook at a time.

So, it seems that it should never be used to specify a path. And so the fact that it had ever been possible was a mistake.

Now I'll try to figure out where the resources path is getting set and see if i can set it relative to the working directory not the notebook directory

[mpacer]

Ok, so it seems that there was an explicit decision to make the relpath relative to the directory of the notebook and not the current working directory: https://github.com/jupyter/nbconvert/commit/e07ef9a67273bc647b2077df5cadf5fd3f9ec182#diff-0ede046bf81229fbf9277b1e4e652252

Additionally #341 suggests it should also be relative to the notebook.

However, build-directory is defined relative to the current working directory — _not_ the notebook.

I'm taking this off of 5.0 because we need to come to a consensus around these definitions and expectations since they've sprung up organically and conflictingly.

[takluyver]

Rats, that does indeed look like we've got ourselves into a confusing corner.

The nbconvert command line interface handles a lot of different things. E.g. if I want to execute a notebook and rewrite the file with output, that's something like jupyter nbconvert --execute --to notebook --inplace .... There's a forest of options, some of which only make sense with other options. Longer term, I think we need to rethink how the CLI works for piecing together the different actions nbconvert can take.

[mpacer]

I feel like we still don't have a consensus around this.

[mpacer]

I can't tell, is this still an issue?

[takluyver]

Specifying --output relative/path still produces the confusing behaviour noted before. I think that we should either make it relative to the cwd, or limit it to just the base filename. Of course, this has now been this way long enough that someone is probably relying on it working the way it currently does.

[Jwink3101]

Has there been any update on this? It has been over a year and you still cannot reliably use nbconvert with subdirectories. Some formats need --output-dir while others do not. And some methods can handle a subdir in the input file while others cannot.

It should be consistent. And the most logical behavior is that it respects paths!

[MSeal]

Moving to 6.0 milestone as the entire build directory process should be considered in how it operates and changing it in 5.x would cause too much disruption.