Nbconvert: `%matplotlib inline` fails when converted to script

[jpeacock29]

This might be known, but didn't see a bug for it. Consider a notebook test.pynb containing the magic %matplotlib inlineand converted to a script test.py:

jupyter nbconvert --to script test.ipynb

test.py now looks like

get_ipython().magic('matplotlib inline')

However, get_ipython() is undefined and the script will be unusable. nbconvert version 5.0.0

[jpeacock29]

I'd be interested in working on resolving this issue, but would need some guidance since this is my first time around this project.

[takluyver]

Magic commands don't work outside of IPython. When we convert a notebook to a .py script, we convert the special syntax to regular Python syntax so that it's a valid .py file, but we can't just remove all the magic commands entirely, because that would change the meaning of the code. If that is what you want, it would be possible (if a bit fiddly) to write a custom exporter to strip all magics, but I don't think we'd make that the default.

Otherwise, I think you just need to manually change those bits after converting it.

[jpeacock29]

Yes, converting magic commands to regular python syntax makes sense to me. I think the bug here is that the python that results from that conversion is not usable. A notebook containing %matplotlib inline will not convert to a usable script; the script will raise an error that get_ipython() is not defined.

As you said, a possible fix would be to strip out the magic commands, but this has obvious shortcomings. Instead, perhaps nbconvert could add an import for get_ipython().

[mpacer]

I'm not sure that would work… but if you try running it using ipython as
your interpreter (ipython your_file.py instead of python your_file.py)
does that work?

On Wed, Dec 28, 2016 at 10:56 Jacob Peacock [e-mail]
wrote:

Yes, converting magic commands to regular python syntax makes sense to me.
I think the bug here is that the python that results from that conversion
is not usable. A notebook containing %matplotlib inline will not convert
to a usable script; the script will raise an error that get_ipython() is
not defined.

As you said, a possible fix would be to strip out the magic commands, but
this has obvious shortcomings. Instead, perhaps nbconvert could add an
import for get_ipython().

—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
https://github.com/jupyter/nbconvert/issues/503#issuecomment-269522800,
or mute the thread
https://github.com/notifications/unsubscribe-auth/ACXg6GPhZ5MlkaQ3DF6ekSDYYJscAGaIks5rMrDmgaJpZM4LWqCw
.

[jpeacock29]

> ipython3 test.py
# [stacktrace]
# UnknownBackend: No event loop integration for 'inline'. 
# Supported event loops are: qt, qt4, qt5, gtk, gtk2, gtk3, tk, wx, pyglet, glut, osx

This basically hits https://github.com/ipython/ipython/issues/9834. We could work around this by calling for a different backend, as get_ipython().magic('matplotlib <backend>'), but that choice of backend is then user dependent. (Perhaps it could be set to the local matplotlib default?)

I could also see the argument that notebooks are intended for GUI display, whereas one would expect a script to run successfully from a terminal, and thus it would be reasonable for nbconvert to remove commands specific to the GUI as part of the conversion process.

[mpacer]

What we could do is just remove the backend, which will set the matplotlib backend to whatever is the default.

[mpacer]

Ok, traced this down.

In nbconvert, this is happening in the ipython2python filter, which is calling IPython.core.inputsplitter.IPythonInputSplitter, which is parsing the ipython logic either through escaped_commands() or assign_from_magic() (not sure which, because those the escape chars are included as noted in a comment for backward compatibility, they're also included first in the list which would mean that they would be hit first if it iterates through the transformers).

But the key is, there's no way to use the current transformation machinery to accomplish this on its own, so we're either going to need to preprocess or postprocess special cases inside ipython2python.

[mpacer]

507 should do this in as efficient a manner as possible, as I was worried about what doing extra processing on every code cell would do to the performance of nbconvert overall on python script exports for large notebooks.

Filtering should only triggered if requested (only the python exporter currently does this, using the no_gui argument to ipython2python) and then defaults to use a precompiled regex (only overwritten if additional backends are requested to be removed).

[Abduoit]

Any solution so far
get_ipython().magic('matplotlib inline') is not converted to .py

[Abduoit]

I write 'exec(%matplotlib inline)' instead of get_ipython().magic('matplotlib inline')

I don't get any error, but there is no output I mean there is no output images.

Anyone can fix it please ???

[hadrien-toma]

Replacing %matplotlib inline in the notebook by 'exec(%matplotlib inline)' works for me, fixing the error and updating outputs as expected. My nbconvert command is:

jupyter nbconvert --template='script_template.tpl' --to python 'notebook.ipynb'

[Abduoit]

Hi @hadrien-toma
I can run properly .ipynb file from jupyter notebook there's no problem
but I want to convert it to .py format, I did the following:-

I used this jupyter nbconvert --to python object_detection_tutorial.ipynb to convert from .ipynb file
to .py file
then I opened the new converted .py file
I removed this get_ipython().magic('matplotlib inline') and I added this 'exec(%matplotlib inline)'
I ran the script by python object_detection_tutorial.py
there is no error in a terminal, everything seems fine, but there is no output bounding images, there is no output at all. !!!!!!!!!!!!! any solution ?

[mpacer]

@Abduoit did you try running ipython object_detection_tutorial.py?

[mpacer]

Also @tacaswell and I discussed this at SciPy. There are unfortunately at least three ways to interpret (4 if you include a somewhat more esoteric use case that involves matplotlib as a means of determining control signals in a human-in-the-loop computational decision process).

What is the actual thing that you want to occur when you want to convert this to a script? Like what is the ideal way this would work out for you?

[Abduoit]

I also tried ipython I got same results, no change.

[mpacer]

What output are you actually looking for as a result of your script?

[Abduoit]

I want to get same output as when I run the script from jupyter notebook, the images should come out with bounding boxes.

[mpacer]

It is literally impossible for it to be the "same" output, because you don't have an interface for displaying it like you do in the notebook. So in what sense do you want it to be the "same"?

How you decide what is the "same" can be done in at least 4 different ways.

[Abduoit]

I was thinking that I can display the output images. Would u plz suggest any code modification that can display the output images.
Thx

[mpacer]

where would you display the output images?

[Abduoit]

any location, let's say come out, or could be automatically saved in the same directory,!

[Abduoit]

I mean, I just want the output images come out once the process in terminal has finished.

[mpacer]

come out? I don't follow.

If you want them to be _saved_ in the same directory, you can just use a different backend. Specifically you could use a non-interactive backend that automatically saves images of particular types: see the tables at the bottom of https://matplotlib.org/faq/usage_faq.html#what-is-a-backend

[mpacer]

but the question is how do they "come out" they need to go somewhere and you need to make a decision about that. If you want them to just be saved use the non-interactive backends.

If you want them to be displayed in the context of running a script, you'll then need to use an interactive backend with some GUI toolkit for displaying the images.
Or, at least that's how I understand it.

Is the previous paragraph more or less correct @tacaswell?

[Abduoit]

sorry for that, I meant
come out = pop up

[mpacer]

But in what context do you want them to "pop up"? This is getting to the point where this is no longer an nbconvert question but a matplotlib question (https://github.com/matplotlib/matplotlib).

Have I provided enough context for you to know how to ask them for what you are looking for?

[Abduoit]

I appreciate your time, I am still new in this area,
I wanted the image as .png or any other type.
I ll check that link, thx

[mpacer]

If you just want the images as png, you could try using the AGG backend with matplotlib.use("AGG") (I'm just guessing since that does png output… I don't know if it will actually work for your use case).

And I'm glad to be able to help. It's just sometimes the most help I can give is to point you to the people who know how to teach you about the details that you need to grok to solve your problem.

Could you come back here once you find a solution that works (and the reason you found that solution to work)?

[Abduoit]

Yeah sure, I ll give it try, then I ll be back here,
thx

[tacaswell]

If you want them to be displayed in the context of running a script, you'll then need to use an interactive backend with some GUI toolkit for displaying the images.
Or, at least that's how I understand it.

Yes, this is correct. Once you involve a GUI backend you now have an event loop (which is to first order a while True: loop) that needs to be run for the figure to be interactive. If you expect an interactive window, then you have to decide if you

• want it to block the execution of the script for each figure you create, requiring it to be closed before the script continues.
• block once at the end and you can see all of the figures created

If you do not want to block for GUI windows than you have to

• save each figure to a file automatically, which requires communicating to the user _what_ they are named
• assume the script is saving figures and just run-through

(These may or may not be the same 4 options I discussed with @mpacer at scipy....)

[li10141110]

@mpacer @Abduoit hi，could you pls tell me what's your results?

[Abduoit]

@li10141110 there's no result, the process finish and there is no output

[FfyFY]

@Abduoit hai, I encounted the same problems recently，is there any new progress in your previous question？ I could not get the output image too .Thank you.

[mzkasu]

@Abduoit @smalllittlewhite Hi, any progress on this? I am also struggling to get an output image. I changed 'matplotlib inline' to 'matplotlib auto' in the code. When running it via ipython, i don't get an error o anything but the images do not show. Eventually i would like to edit the code and then place the 'result' images into a new folder showing what its detected.

[rgerkin]

Here's a way to handle the magics, assuming your converter is turning magic commands into get_ipython().run_line_magic commands (which regular Python cannot execute) rather than scrubbing them entirely. Parse the exported Python file into a list of lines with e.g.:

with open('/path/to/exported/python/file') as f:
    code = f.read()
code = code.split('\n')

and then apply this to each line in code using:

def strip_line_magic(line,magics_allowed):
        matches = re.findall("run_line_magic\(([^]]+)", line)
        if matches and matches[0]: # This line contains the pattern
            match = matches[0]
            if match[-1] == ')':
                match = match[:-1] # Just because the re way is hard
            magic_kind,stripped = eval(match)
            if magic_kind not in magics_allowed:
                stripped = "" # If the part after the magic won't work, just get rid of it
        else:
            printd("No line magic pattern match in '%s'" % line)
            stripped = line
        return stripped
````

For example, if we make a list of magics that are likely to precede an otherwise-functioning line of code (e.g. %time, %timeit, etc.) and exclude from that list things that aren't expected to precede valid code (e.g. %matplotlib), then we can scrub the magics that won't work in Python and keep the ones where the magic is just incidental.  

For example:  

magics_allowed = ['time']
example1 = "get_ipython().run_line_magic('matplotlib', 'inline')" # 'inline' is not valid code without magic
example2 = "get_ipython().run_line_magic('time', 'x = np.random.rand(10000)'" # 'x = ...' is valid code without magic
strip_line_magic(example1,allowed) # Returns '"" since this line must be purged
strip_line_magic(example1,allowed) # Returns "x = np.random.rand(10000)", the code we want to run
```

[johndpope]

slightly off topic - but hypothetically - using iterm - someone could rig up the terminal to actually show image at the command line. Here's the heavy lifting already done - just need to call out from python to command line https://github.com/johndpope/terminal-image-cli
Examples
$ image unicorn.jpg
$ cat unicorn.jpg | image

[shamik111691]

@Abduoit did you figure this out?

[git-ness]

I was having this issue in PyCharm. The fix for me was I didn't have the matplotlib package installed. Not sure if this applies to anyone else but figured I'd mention it in case it helps.

[Skitionek]

You might as well spawn InteractiveShell class to wrap ipython 'magic' calls.
I have done it by adding cell to notebook which will run in case of execution outside of ipython:

if '_' not in globals():
    print("Running outside of ipython")
    from IPython import InteractiveShell as get_ipython

not sure about matplotlib but it works like a charm for command line calls (actually executes them).

[t-makaro]

I would have proposed added an "ipython" exporter that doesn't do anything with magic commands and creates a .ipy file, and then the python exporter could just rip out the magic commands. However, @Skitionek's workaround looks even better to me though I haven't tried it. This might work best if we can detect if magics are used, and then add the code block to the start if needed.