Mu: pip doesn't work on first run.

[ntoll]

FYI @tjguk - I'm going to look into this right now but may drop it if it's a rabbit hole. Reporting for visibility, but I hope the fix is easy (see below).

• What you were trying to do

Start Mu for the first time and then install a new package into the clean venv.

• What steps you took to make this happen,

From a clean start, run Mu and allow it to set up a new venv and other assets. Then, without restarting Mu, try to install a package from the admin screen in Python3 mode. Screen-grab of this happening shown below.

• What you expected to happen,

I should be able to see the usual installation log and pip should work as expected.

• What actually happened,

The installation log is just blank. If I restart Mu, it works as expected.

• Why this difference is problematic (it may not be a bug!),

It should work first time... ;-)

• Technical details like the version of Mu you're using, your OS version and
  other aspects of the context in which Mu was running.

Latest master. I think something like the path to Python or pip in the venv is not being set properly on first run such that the admin based install packages widget cannot work.

[ntoll]

OK... it looks like it's because the settings.json and venv.json files are not created until the editor is closed for the first time. I'll do a quick fix and push to master so this bug isn't in beta.2. :+1:

[ntoll]

OK... that was a red herring. Still trying to figure out what's going on.

[ntoll]

Parking for the moment.

TL;DR I don't think the slots are created in the right way here:

https://github.com/mu-editor/mu/blob/master/mu/interface/dialogs.py#L647

I suspect some state in the virtualenvironment.py module isn't set on first run. I.e. the venv singleton is in a pre-ensure-and-created state, when it should really be re-configured post-creation. I did try calling relocate but that didn't fix things.

@tjguk any ideas..?

[ntoll]

I'll mention this as a known bug in the release notes.

[tjguk]

Curiouser and curiouser...

Everything seems to be in place in terms of slots going to the right places etc. (And, evidently, it all works fine on the second run). I ran with a run_blocking command, probably expecting that some venv/pip-related error was being swallowed and the job failing to progress. But that produced perfectly good pip output, albeit at the end of the run, rather than as it progresses.

(I'm also rather surprised that this hasn't shown up in ad hoc testing previously. The sequence of events which is now failing is typically what I'd have done to test that the pip-install functionality was working. Possible that I haven't in fact done this, but also that something to do with the splash screen functionality is interfering on the first run. Goodness knows why...)

I'll keep looking. I can probably produce a workaround by running pip blocking (which seems to work). For most cases you'll hardly notice the difference in timing.

[tjguk]

Just a couple more datapoints: even if I implement a "blocking" version of the unblocked run, by basically running the code to completion and then calling the output` slot, it still doesn't produce the output, all it does complete and the "FINISHED" text appears on the text display.

Also, to eliminate the possibility, I tried opening the editor as new, doing no installation, closing it, opening it again and pip-installing -- and that works. So it's not just the first time you try; it's the first time immediately after a new setup.

[tjguk]

(Apologies for the stream of consciousness; I only have bits and pieces of time available so I'm leaving myself a trail to pick up later...)

The obvious difference is that the first time we see the Qt thread/parent warning which doesn't appear the second time.

2021-03-14 18:21:36,186 - mu.virtual_environment:199(run) DEBUG: About to run blocking: C:\Users\Tim.Golden\AppData\Local\python\mu\mu_venv-38-20210314-182113\scripts\pip.exe, ['list', '--disable-pip-version-check'], 30.0
2021-03-14 18:21:37,096 - mu.virtual_environment:608(install_jupyter_kernel) INFO: Installing Jupyter Kernel: "Python/Mu (mu_venv-38-20210314-182113)"
QObject: Cannot create children for a parent that is in a different thread.
(Parent is Process(0x1dc4c1fbad0), parent's thread is QThread(0x1dc4a3095c0), current thread is QThread(0x1dc4d78fe00)
2021-03-14 18:21:38,202 - mu.virtual_environment:440(ensure_and_create) DEBUG: Checking virtual environment; attempt #2.

[ntoll]

Well, that warning always feels a bit of a "niggle" to me... if we can squash it I'll feel like I've balanced my parentheses. My kids are making this evening's family (mothers' day) dinner, so I'll try to get to the cause (if not the fix) of the problem in the next hour or so I have left to me.

[tjguk]

So I think I've found the cause of the Qt warning... which has now thrown up, for the first time, a different Qt warning when we try to call pip install.

[tjguk]

Basically, virtual_environment.Process._set_up_run is what actually creates the QtProcess, passing in the Process object (which is a subclass of QObject as the "parent"). For reasons I haven't looked into yet, that's what causes Qt to complain -- although I don't know why it does that only this time round!

Running again, that now gives us:

QObject::connect: Cannot queue arguments of type 'QProcess::ExitStatus'
(Make sure 'QProcess::ExitStatus' is registered using qRegisterMetaType().)

in the Process.run method. About to look into that now.

[ntoll]

@tjguk hah.... I just go to that method. Since you're on it, I'll switch to something else (the release / changes docs for tomorrow).

[ntoll]

At risk of muddying waters... the QThread object of the Process object is different when the Jupyter kernel installation happens. From my pdb:

$ ./run.py 
No settings file found at /home/ntoll/.local/share/mu/venv.json; skipping
> /home/ntoll/src/mu-editor/mu/mu/virtual_environment.py(114)_set_up_run()
-> self.process = QProcess(self)
(Pdb) self.thread()
<PyQt5.QtCore.QThread object at 0x7f4d4223e700>
(Pdb) cont
> /home/ntoll/src/mu-editor/mu/mu/virtual_environment.py(114)_set_up_run()
-> self.process = QProcess(self)
(Pdb) self.thread()
<PyQt5.QtCore.QThread object at 0x7f4d4223e700>
(Pdb) cont
> /home/ntoll/src/mu-editor/mu/mu/virtual_environment.py(114)_set_up_run()
-> self.process = QProcess(self)
(Pdb) self.thread()
<PyQt5.QtCore.QThread object at 0x7f4d4223eaf0>

Not sure why the associated thread is different the third time.

[tjguk]

Ah; that's interesting. I doubt I'll have time to continue the pursuit tonight. I'll try to pick up the threads [*] again tomorrow

[*] No pun intended

[ntoll]

I also notice (and probably not relevant, but just saying anyway) that the first two commands are pip whereas the problematic third run is python -m ipython etc... Clutching at straws here, but could this be significant..? I wonder what happens is we python -m pip install blah... for the first two commands..?

[tjguk]

Thanks, @ntoll -- there might be something in that. Both ultimately resolve to one of the run / run_blocking methods of the virtual_environment.Process object. When I get a chance, I'll see what happens if I stub out the Kernel creation

[ntoll]

We're both "old hands" at this, so I like how we both acknowledge that it probably shouldn't have any impact, but are open to the potential that it does in some yet-to-fathom manner.

[tjguk]

Key comment from https://stackoverflow.com/questions/17083379/qobjectconnect-cannot-queue-arguments-of-type-int:

"If Qt is trying to queue the arguments that means that the connection is between threads. This will not work for non-const references."

So we're still implicitly using threads, at least for the initial run. Need to understand why

[tjguk]

... and also useful (since we're trying to append to a QPlainText control via a signal/slot):

https://stackoverflow.com/questions/2104779/qobject-qplaintextedit-multithreading-issues

[aroberge]

I'll mention this as a known bug in the release notes.

I did not see any mention of this bug (which I encountered) when reading https://codewith.mu/en/howto/1.1/install_windows (or the Mac version). Perhaps it should be added as a note. After restarting Mu, everything worked as expected (save for the help button [french ui] which leads to a non-existent page.)

On the positive side, the "admin" interface with only three tabs is simpler than what I used before (1.1.0.alpha2) and friendly still works :-)

Thank you all for your hard work!

[tjguk]

Take the Splash Screen out of the mix, and the problem goes away! (Perhaps unsurprising as the Splash Screen mechanism is the only instance of explicit QThreading we have). I'll come back to this later

[tjguk]

Ok; I've got to dash now, but I think I've got it. (Memo for later: the virtual_environment.Pip object is initially created from the SplashScreen worker thread if the venv doesn't originally exist via relocate.