Octoprint: [Brainstorming] Get octoprint running on python3

[razerraz]

Maybe it's not right place for that, feel free to move/delete this
I manage to get octoprint running on python 3.6
For sure, it's far from running completly, but the core is loading, connection with printer ok, flask pages and sockjs push ok, base plugins ok, wizard ok.
It's mainly type conversion hacks, import changes and fighting with unmaintained sockjs-tornado that seems to be the main migration issue here. Nevertheless socket is partialy running too, with my own hacks in the module. I just have issues with crashs in the asyncio loop without any log at all, as I have with other asyncio projects.
It's mainly a game for me and a way to learn things about consequent basecode migration, but maybe my work can help here. Feel free to request all my changes via a fork or a pull request, keeping in mind that the merge will be a mess
Cheers

[GitIssueBot]

Hi @razerraz,

It looks like there is some information missing from your bug report that will be needed in order to solve the problem. Read the Contribution Guidelines which will provide you with a template to fill out here so that your bug report is ready to be investigated (I promise I'll go away then too!).

If you did not intend to report a bug but wanted to request a feature or brain storm about some kind of development, please take special note of the title format to use as described in the Contribution Guidelines.

Please do not abuse the bug tracker as a support forum - that can be found at discourse.octoprint.org. Go there for any kind of issues with network connectivity, webcam functionality, printer detection or any other kind of such support requests or general questions.

Also make sure you are at the right place - this is the bug tracker of the official version of OctoPrint, not the Raspberry Pi image OctoPi nor any unbundled third party OctoPrint plugins or unofficial versions. Make sure too that you have read through the Frequently Asked Questions and searched the existing tickets for your problem - try multiple search terms please.

I'm marking this one now as needing some more information. Please understand that if you do not provide that information within the next two weeks (until 2018-07-23 20:00 UTC) I'll close this ticket so it doesn't clutter the bug tracker. This is nothing personal, so please just be considerate and help the maintainers solve this problem quickly by following the guidelines linked above. Remember, the less time the devs have to spend running after information on tickets, the more time they have to actually solve problems and add awesome new features. Thank you!

Best regards,
~ Your friendly GitIssueBot

PS: I'm just an automated script, not a human being, so don't expect any replies from me :) Your ticket is read by humans too, I'm just not one of them.

[tedder]

See #1416 . Perhaps you could make PRs for the good bits of your conversion and not the hacks.

[razerraz]

Well, for what I've seen so far, if the idea behind is to make something working both in python 2.7 & 3.x, the task will be very hard and bothering.
Perhaps having a python3 specific branch is the best way to go, imho, then I can do some PR

[foosel]

I took the liberty to turn this into a Brainstorming ticket.

A python 3 specific branch is out of the question I fear. I already have way too much overhead as is, I seriously don't want to have to start doing everything twice across not only one but up to three concurrent development branches (devel, maintenance, staging/maintenance).

The goal is to make this whole code base compatible to 2 and 3, and vendor bundle and adjust any third party dependencies that don't support this if push comes to shove.

sockjs-tornado will already be vendor bundled starting with 1.3.9 anyhow since it appears to be unmaintained and there are some bug fixes available that are still not merged to the official release version, so I went that way.

The thing I'm most worried about myself is indeed unicode vs str and that will probably take the longest time to get right.

[tedder]

Yeah, the str/bytes thing ends up being chased a lot. 2to3 isn't the prettiest for converting either.

Do you plan on cutting over or supporting both?

[foosel]

I'll have to support both for a while until all the plugin authors have had a chance to convert their plugins as well. Another issue is finding a migration path for all those ancient OctoPi installs out there that will have to be switched to Python 3 as well then. Although i'm leaning towards "upgrade your images please" in that case.

[tedder]

I'd prefer a short "support both" period- maybe only 1.5, and then 1.6 fully deprecates it. It was annoying to write Ansible plugins that had to support everything from py 2.6 to 3.x.

Just my vote, though.

[foosel]

The current goal is to get both working with 1.4.0 and then drop support for 2 in 1.5.0. Maybe even declare that 2.0 due to semver vs plugins. I'm not fully sure about that yet.

I'm also not looking forward to this at all, especially since I already have enough on my plate as is, but well, no choice. Big bang would be easiest for me but suck for most users.

[razerraz]

I have time right now and continue to sort everything out running on py3, without really thinking deeply on having the thing py2 compliant as well. Will come...
Right now, it's running on a venv, all tests are passing ok, just some curious things with decorators and functools.
Of course, the list of changes is huge and affects sources and tests, but a lot of things can be done right now with very limited risk :

• Use int argument for yaml indent attribute
• Wrap every map/filter/... into a list object
• Use six facilities for all isinstance() involving unicode

I will do one proper PR for every items mentioned, and one another fork with all my changes, the best horror movie you can watch I guess

[foosel]

Use six facilities for all isinstance() involving unicode

Careful, there's already a dependency on python-future for stuff like this.

[smurfix]

@razerraz are you still working on porting to Py3?

[razerraz]

@smurfix right now I'm quite busy with other things, but I will soon hopefully
There is a branch here : https://github.com/razerraz/OctoPrint/tree/Py3_devel that is running on py2 + py3 if I remember well, and if it can help right now

[foosel]

There's also a ton of PRs open regarding Py3 support that I really need to get around to review and merge. Regular maintenance and life just keep getting in the way 😒

[smurfix]

Yeah, life's like that. I just noticed these PRs … I started merging @razerraz 's branch up, can probably finish that this weekend. Then I could do the same thing with @ByReaL 's PRs, merge them all into a different branch, and cross-check the two. That should help.

[ByReaL]

i'll after you merge mines in a branch i need probably few days to do some further testing and changesi would like to install that branch on py 2 and 3 and start an actual 3D print and play with the UI a bit 
On Friday, January 25, 2019, 4:08:24 AM PST, Matthias Urlichs [e-mail] wrote:

Yeah, life's like that. I just noticed these PRs … I started merging @razerraz 's branch up, can probably finish that this weekend. Then I could do the same thing with @ByReaL 's PRs, merge them all into a different branch, and cross-check the two. That should help.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub, or mute the thread.

[smurfix]

I'm working my way through the difference between @razerraz 's patches, @ByReaL 's patches, and @foosel 's maintenance work. Will finish tomorrow (probably) and then do some light testing.

[ByReaL]

ok, i'll set some time aside next weekend to submit the rest of the changes, and ensure that i can 3D print something using both py2 and py3let me know the name of the branch that should be used for this.
On Sunday, January 27, 2019, 1:20:28 PM PST, Matthias Urlichs [e-mail] wrote:

I'm working my way through the difference between @razerraz 's patches, @ByReaL 's patches, and @foosel 's maintenance work. Will finish tomorrow (probably) and then do some light testing.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub, or mute the thread.

[smurfix]

I'm through with the basic integration stuff, though I'm sure I missed a few things (all still untested).

My branch is at github.com/smurfix/OctoPrint (py23 branch) though I'll probably rebase it to keep the changes reasonably clean and separate. I'll report back when I'm reasonably done with that, most likely this evening.

[smurfix]

OK, tests for Python 2.7 and 3.6 pass. This is not done (I didn't yet try to print anything!) but I will no longer rebase the branch.

@foosel Please take a look. Maybe we could continue this in the main repository.

[smurfix]

NB: I probably made some mistakes while cleaning up the merge conflicts, so my next task will be to merge @foosel 's maint and dev branches (unless she beats me to it … hint, hint ;-) and then go through the diff from that to py23 with a hopefully-sufficiently-fine comb.

[razerraz]

As far as I remember what I've done, most of the painful job will be having install script working py2+3, considering all the dependency versions differences.
The other big thing is plugins. For sure, maintainers of them will have some changes to do, but it have to be trivial. I fear that's not the case in my branch since I never spend time to consider it.
At the end, a major refactoring should be done to avoid horrors like _list(map())_, ie integrate iterations in the chain.
Glad to see this kind of activity here, grats

[foosel]

@smurfix Huge thanks for the work you put in there, I really appreciate it!

For the record, we are talking about https://github.com/smurfix/OctoPrint/tree/py23, correct? Which of the Py3 tagged PRs did you merge now, all of them? If so I'd suggest when you call that good to go we merge that into a dedicated feature branch on the main repository and then take it from there. That way we finally have one single branch to put through final tests (I have to admit that one of the reasons the reviews take me so long is that I simply keep losing track of which covers what - the other is the usual project management madness) and then can merge that in one go to devel and at that point travis should also be adjusted to test against both python versions.

[foosel]

The other big thing is plugins. For sure, maintainers of them will have some changes to do, but it have to be trivial.

It might indeed make sense to introduce a new flag about python compatibility in the repository listings. That way only such plugins will be allowed to be installed through the plugin manager that are known to be py3 compatible.

Next big update after 1.4.0 will then go full py3 (which I'll probably have to call 2.0.0 instead of 1.5.0 because that is a strong case of backwards incompatibility).

most of the painful job will be having install script working py2+3, considering all the dependency versions differences.

I hope that it won't be that much that needs to be adjusted there, since last I checked most or even all dependencies were already py3 compatible - did I miss something?

[smurfix]

Yes, I merged all of the PRs. There were some conflicts, which is why IMHO this kind of work is best done in one line of development with one type of change per commit, and frequent rebasing - because you're sure to have missed something (which is what I did after mostly-octopus-merging all the py23 pull requests).

https://github.com/smurfix/OctoPrint/tree/py23 is correct. Yes, we do need a compatibility level thing, preferably one that we can read without importing the module in question. I'd use simple (i.e. ' eval' able) OCTOPRINT_MIN_REL and OCTOPRINT_MAX_REL integers, or maybe semver-style triples. Then again, if even Firefox with its insanely complex interface doesn't need more than a simple integer, then so do we. ;-)

There have been reasonably few list(map()) instances. Again, I'm sure I've missed a few …

WRT dependencies: Four or five of then required an uptick, according to @razerraz . See the current version of setup.py in my repository. I'll clean that up with another change shortly, as requiring different dependencies depending on Py2 vs. Py3 doesn't seem particularly maintainance friendly to me; there's the "future" module which is Py2 only, but that's it IMHO.

[razerraz]

I remember having to get some of the dependencies up to date, that was getting stuck for py2. Maybe things have changed in recent version.
And for iterables wrapped as list, you can apply it on all functional stuff : filter, reduce... And by the way, even if the amount is not so huge (I doubt about it too), refactoring changes seems to me huge...

[foosel]

Yes, we do need a compatibility level thing, preferably one that we can read without importing the module in question.

I extract the name and version from installed plugins without actually importing or otherwise running them in any way through the AST of the module (I introduced that after I caught some plugins happily jumping into action just by importing them, allowing them to partially bypass manual disabling or blacklist). I guess something similar could be done here, with an assumed default of "py27 only" if not overwritten by the plugin author.

But I was actually talking about the repository and plugin manager as a first step, where we already check against OctoPrint versions & underlying operating system before an installation is even attempted let alone the plugin detected and its own metadata fetched.

[smurfix]

Status: I went over the source with a fine-toothed comb and list()ed all instances of map/filter/keys/values/items that needed it (and un-list()ed a bunch that did not).

However, currently the server goes into an endless loop when I access the start page, for both Py2 and Py3. I suspect that I broke something when I merged the dev and main branches.

@foosel :: could you please merge maint into dev and get it to work? I will then do something sensible with the result so that (a) it's Py3 compatible and (b) the history isn't excessively convoluted, given the fact that I already did an obviously-somewhat-broken merge of these two branches while merging the Py3 PRs.

[foosel]

@smurfix on it

[foosel]

@smurfix Merge is complete and working in principle, however I see message duplication on the websocket and am not yet sure where that is coming from. Have to investigate further, might take until tomorrow. I don't want to push the current semi broken state since I know that a (very small) number of people use devel.

[smurfix]

So push it to another branch …

[foosel]

See https://github.com/foosel/OctoPrint/tree/dev/merge_201901

[foosel]

Strike that, finally found and fixed the duplication bug, merge result is now available on devel

[smurfix]

I still have a problem: With an empty .octoprint and a newly-minted Python2.7 virtualenv, the browser gets into an endless loop where it briefly shows the main screen, then the spinner, repeat ad infinitum.

This makes testing the Py3 version a bit difficult – it does the same thing, but that's hardly helpful at this point …

2019-01-31 20:21:52,690 - octoprint.server.util.sockjs - INFO - New connection from client: 10.107.2.5
2019-01-31 20:21:52,903 - octoprint.server.util.flask - INFO - Passively logging in user None from 10.107.2.5
2019-01-31 20:21:53,894 - octoprint.server.util.sockjs - INFO - Client connection closed: 10.107.2.5
2019-01-31 20:21:54,968 - tornado.access - WARNING - 403 GET /api/timelapse?unrendered=true (10.107.2.5) 3.08ms
2019-01-31 20:21:56,976 - octoprint.server.util.sockjs - INFO - New connection from client: 10.107.2.5
2019-01-31 20:21:57,023 - octoprint.server.util.flask - INFO - Passively logging in user None from 10.107.2.5
2019-01-31 20:21:58,037 - octoprint.server.util.sockjs - INFO - Client connection closed: 10.107.2.5

(repeats)

Browser console (copy+paste):

Navigated to http://dev.vm.smurf.noris.de:50080/#temp
User logged out packed_core.js:3286:25
... binding done packed_core.js:17214:13
Application startup complete packed_core.js:17226:13
Less has finished and no sheets were loaded. less.min.js:13:12757
Password fields present on an insecure (http://) page. This is a security risk that allows user login credentials to be stolen.[Learn More]
dev.vm.smurf.noris.de:50080
Password fields present on an insecure (http://) page. This is a security risk that allows user login credentials to be stolen.[Learn More] dev.vm.smurf.noris.de:50080
Starting dependency resolution... packed_core.js:16853:9
... dependency resolution done packed_core.js:16963:9
Initial application setup done, connecting to server... packed_core.js:17265:9
Connected to the server packed_core.js:14953:9
Finalizing application startup packed_core.js:17243:13
Going to bind 39 view models... packed_core.js:17122:13
Did not bind view model SoftwareUpdateViewModel to target #softwareupdate_confirmation_dialog since it does not exist packed_core.js:17175:33
Did not bind view model SoftwareUpdateViewModel to target #wizard_plugin_softwareupdate since it does not exist

[foosel]

Huh... I'll take a look tomorrow, maybe the empty .octoprint folder is the key.

[ByReaL]

Will try to have a look this weekend also.

Thanks,
Mircea Dan

On Jan 31, 2019, at 11:36 AM, Gina Häußge [e-mail] wrote:

Huh... I'll take a look tomorrow, maybe the empty .octoprint folder is the key.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub, or mute the thread.

[foosel]

Good news, with an empty .octoprint folder I can reproduce this looping error.

[foosel]

Solved it by removing the forcelogin plugin. We don't need it on devel because its function is pretty much superseded by the granular permission system, and its logic that's still based on the maintenance application initialization was causing the loop. Then I fixed the wizard which was broken by a misinterpretation on how deprecation wrapping in parent classes works. And then I fixed a bug in maintenance that caused the webcam to not be displayed and merged that into devel too. And now I hope the worst problems should be fixed. Phew.

[smurfix]

That's great news.

Next: I can simply merge devel into py23, which gives us a rather convoluted parallel history of non-working bits and pieces (not cool if somebody ever needs to do a "git bisect", but otherwise mostly harmless), or I can do half a day's work of splitting it all up into a new patchset. If you'd like to see the latter, not a problem, I'm home sick anyway and this is mostly what my brain is capable of at the moment 😔 but if nobody's going to need the work anyway I could do something mire creative … like actually design the bits and pieces I want to make with OctoPrint in the first place. 🤖

[foosel]

I think at this point the history around this point in time is pretty much 😲 anyhow, and it wouldn't be the first time a git bisect can run into such situations in this project. So I'd say: merge. Or rebase if that's an option. But I wouldn't go through the hassle of splitting everything up yet again.

Also: get well soon! And I'm frankly astonished that you feel capable of merging horror in that state, when I'm sick these days I try to not touch any code anymore, past experience has shown me that it rarely ends well 😅

[smurfix]

You missed something:
git grep DummyUser

[smurfix]

anyway, disregarding that, the current py23 branch (which I just pushed), while looking reasonably sane when looking at the diff, doesn't work with Py3 yet. Some more debugging and cleanup is necessary.

I'm of two minds about from future import unicode_literalsing the whole code base, but on the whole I think that we might progress faster if we do that.

[smurfix]

Did that. Also did a bit of code cleanup. Initial startup now works, I also tested uploading. I didn't yet test actual printing, that'll happen tomorrow.

[smurfix]

Branch status: communication with the printer works. I didn't actually print anything yet and will undoubtedly hit a few unicode-vs-bytes corners that have been overlooked, but I consider the work to be 95% finished and there should be no major issues left.

[foosel]

You missed something:
git grep DummyUser

Should be fixed now.

I'm of two minds about from future import unicode_literalsing the whole code base, but on the whole I think that we might progress faster if we do that.

I fear you are right. I don't like it very much, but on the other hand it should indeed allow for faster progress.

I consider the work to be 95% finished and there should be no major issues left.

Sounds good! 👍 When you think it feels ready, can you create a consolidated PR? I'll then give that a test drive against a printer as well and if things look good we can merge (and if not, fix stuff). And then I have to think of a good strategy to avoid reintroducing incompatibility problems by merges from maintenance or other branches.

And a huge thank you for putting in that work, that really put a huge load off my shoulders!

[smurfix]

I'll proceed until I have a working test print; should be tomorrow at the latest.

We probably should create a test case that actually runs a server and uploads+prints a test file to the virtual printer. That should catch many regressions.

[smurfix]

See #3032.

[smurfix]

OK. As the last step (so far …) I ran flake8 on the Py23 branch, which found a bunch of interesting errors – I recommend to add a run of "flake8 --ignore=D,E,W src tests setup.py" to the testsuite.

Printing and manual controls works for me. I found a problem with retransmission that I think is fixed (can't reproduce because my printer isn't that flaky). Somebody needs to run tests with an SD-card-equipped printer; mine isn't and that's the main part that might require some more work.

Barring strong opinions from @foosel 😇 I declare this branch to be in go-forth-and-get-it-merged quality.

[foosel]

I found a problem with retransmission that I _think_ is fixed (can't reproduce because my printer isn't _that_ flaky).

The virtual printer is set up to simulate flakyness at line 100, so if the issue you observed was actually a problem with the resend handling itself and not something only observable against a real serial port you should be able to verify any fixes against that. I can also take a look, but for that I need more info on what kind of error you encountered :)

Somebody needs to run tests with an SD-card-equipped printer; mine isn't and that's the main part that might require some more work.

Virtual printer also has SD support.

Barring strong opinions from @foosel 😇 I declare this branch to be in go-forth-and-get-it-merged quality.

I'm setting aside some time tomorrow for a deep dive and test, but based on what I've seen so far it looks good 😄👍

One question, in the PR you mentioned flake8 now running through (with the mentioned ignores) - I guess it would also make sense to just add that to .travis.yml to run on every build going forward, any objections to that?

[smurfix]

One question, in the PR you mentioned flake8 now running through (with the mentioned ignores) - I guess it would also make sense to just add that to .travis.yml to run on every build going forward, any objections to that?

No, of course not. Do it.

NB: I did a couple of virtual printer tests but the emulated SD card directory was empty. My bug, your bug, or not implemented?

[foosel]

My bug, your bug, or not implemented?

Last I checked it did work here. I also have files in there (some I streamed, some I copied manually). I cannot test right now against devel since I'm fighting with an addition to the permission system made necessary by removing the forcelogin plugin from 1.4.0, and things are currently not in a good state ;)

[smurfix]

OK, I'll look into it.

[foosel]

Print under py3 went through without hiccups, print under py2 is currently running and so far looking normal too. So the most important stuff appears to be working. Unless anything horrible gets unearthed by someone over night, I'll merge #3032 tomorrow.

@ByReaL you wanted a heads-up about this, this is the heads-up.

[ByReaL]

thank you very much On Wednesday, February 6, 2019, 10:02:22 AM PST, Gina Häußge [e-mail] wrote:

Print under py3 went through without hiccups, print under py2 is currently running and so far looking normal too. So the most important stuff appears to be working. Unless anything horrible gets unearthed by someone over night, I'll merge #3032 tomorrow.

@ByReaL you wanted a heads-up about this, this is the heads-up.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub, or mute the thread.

[razerraz]

I'm currently testing devel branch with a py3 venv : seems to work fine, what a tremendous job you've done on a small amount of time it's amazing. The speed gain is noticeable on the small nanopi board I'm using.
My test case is trying to install third party plugins on a fresh install, and I've already found a bug that prevent installing them, and sadly one of my PR is the cause. Fix can be found here :
https://github.com/razerraz/OctoPrint/commit/d269c361a4e4f90fbb35afa4948f8e88fe5fecd0

Otherwise, I ran into issues involving software update plugin, permissions, json encoding and got lost. The main exception was ::

Feb 09 17:55:34 3dprinter octoprint[4761]: [2019-02-09 17:55:34,119] ERROR in app: Exception on /plugin/softwareupdate/check [GET]
Feb 09 17:55:34 3dprinter octoprint[4761]: Traceback (most recent call last):
Feb 09 17:55:34 3dprinter octoprint[4761]:   File "/home/razer/OctoPrint-3c5aa60/venv/lib/python3.7/site-packages/OctoPrint-1.4.0.dev2000+g3c5aa60d-py3.7.egg/octoprint/server/util/flask.py", line 1567, in default
Feb 09 17:55:34 3dprinter octoprint[4761]:     return JsonEncoding.encode(obj)
Feb 09 17:55:34 3dprinter octoprint[4761]:   File "/home/razer/OctoPrint-3c5aa60/venv/lib/python3.7/site-packages/OctoPrint-1.4.0.dev2000+g3c5aa60d-py3.7.egg/octoprint/util/json/__init__.py", line 31, in encode
Feb 09 17:55:34 3dprinter octoprint[4761]:     raise TypeError
Feb 09 17:55:34 3dprinter octoprint[4761]: TypeError
Feb 09 17:55:34 3dprinter octoprint[4761]: During handling of the above exception, another exception occurred:
Feb 09 17:55:34 3dprinter octoprint[4761]: Traceback (most recent call last):
Feb 09 17:55:34 3dprinter octoprint[4761]:   File "/home/razer/OctoPrint-3c5aa60/venv/lib/python3.7/site-packages/Flask-0.12.4-py3.7.egg/flask/app.py", line 1982, in wsgi_app
Feb 09 17:55:34 3dprinter octoprint[4761]:     response = self.full_dispatch_request()
Feb 09 17:55:34 3dprinter octoprint[4761]:   File "/home/razer/OctoPrint-3c5aa60/venv/lib/python3.7/site-packages/Flask-0.12.4-py3.7.egg/flask/app.py", line 1614, in full_dispatch_request
Feb 09 17:55:34 3dprinter octoprint[4761]:     rv = self.handle_user_exception(e)
Feb 09 17:55:34 3dprinter octoprint[4761]:   File "/home/razer/OctoPrint-3c5aa60/venv/lib/python3.7/site-packages/Flask-0.12.4-py3.7.egg/flask/app.py", line 1517, in handle_user_exception
Feb 09 17:55:34 3dprinter octoprint[4761]:     reraise(exc_type, exc_value, tb)
Feb 09 17:55:34 3dprinter octoprint[4761]:   File "/home/razer/OctoPrint-3c5aa60/venv/lib/python3.7/site-packages/Flask-0.12.4-py3.7.egg/flask/_compat.py", line 33, in reraise
Feb 09 17:55:34 3dprinter octoprint[4761]:     raise value
Feb 09 17:55:34 3dprinter octoprint[4761]:   File "/home/razer/OctoPrint-3c5aa60/venv/lib/python3.7/site-packages/Flask-0.12.4-py3.7.egg/flask/app.py", line 1612, in full_dispatch_request
Feb 09 17:55:34 3dprinter octoprint[4761]:     rv = self.dispatch_request()
Feb 09 17:55:34 3dprinter octoprint[4761]:   File "/home/razer/OctoPrint-3c5aa60/venv/lib/python3.7/site-packages/Flask-0.12.4-py3.7.egg/flask/app.py", line 1598, in dispatch_request
Feb 09 17:55:34 3dprinter octoprint[4761]:     return self.view_functions[rule.endpoint](**req.view_args)
Feb 09 17:55:34 3dprinter octoprint[4761]:   File "/home/razer/OctoPrint-3c5aa60/venv/lib/python3.7/site-packages/OctoPrint-1.4.0.dev2000+g3c5aa60d-py3.7.egg/octoprint/server/util/flask.py", line 1242, in decorated_view
Feb 09 17:55:34 3dprinter octoprint[4761]:     return func(*args, **kwargs)
Feb 09 17:55:34 3dprinter octoprint[4761]:   File "/home/razer/OctoPrint-3c5aa60/venv/lib/python3.7/site-packages/OctoPrint-1.4.0.dev2000+g3c5aa60d-py3.7.egg/octoprint/access/permissions.py", line 158, in _decorated
Feb 09 17:55:34 3dprinter octoprint[4761]:     rv = f(*args, **kw)
Feb 09 17:55:34 3dprinter octoprint[4761]:   File "/home/razer/OctoPrint-3c5aa60/venv/lib/python3.7/site-packages/OctoPrint-1.4.0.dev2000+g3c5aa60d-py3.7.egg/octoprint/plugins/softwareupdate/__init__.py", line 586, in check_for_update
Feb 09 17:55:34 3dprinter octoprint[4761]:     unless=lambda: force)(view)()
Feb 09 17:55:34 3dprinter octoprint[4761]:   File "/home/razer/OctoPrint-3c5aa60/venv/lib/python3.7/site-packages/OctoPrint-1.4.0.dev2000+g3c5aa60d-py3.7.egg/octoprint/server/util/flask.py", line 1089, in decorated_function
Feb 09 17:55:34 3dprinter octoprint[4761]:     response = f(*args, **kwargs)
Feb 09 17:55:34 3dprinter octoprint[4761]:   File "/home/razer/OctoPrint-3c5aa60/venv/lib/python3.7/site-packages/OctoPrint-1.4.0.dev2000+g3c5aa60d-py3.7.egg/octoprint/plugins/softwareupdate/__init__.py", line 544, in view
Feb 09 17:55:34 3dprinter octoprint[4761]:     dict(name=gettext("setuptools"), current=self._environment_versions.get("setuptools", "unknown"), minimum=MINIMUM_SETUPTOOLS)])))
Feb 09 17:55:34 3dprinter octoprint[4761]:   File "/home/razer/OctoPrint-3c5aa60/venv/lib/python3.7/site-packages/OctoPrint-1.4.0.dev2000+g3c5aa60d-py3.7.egg/octoprint/server/util/flask.py", line 277, in fixed_jsonify
Feb 09 17:55:34 3dprinter octoprint[4761]:     dumps(data, indent=indent, separators=separators) + '\n',
Feb 09 17:55:34 3dprinter octoprint[4761]:   File "/home/razer/OctoPrint-3c5aa60/venv/lib/python3.7/site-packages/Flask-0.12.4-py3.7.egg/flask/json.py", line 167, in dumps
Feb 09 17:55:34 3dprinter octoprint[4761]:     rv = _json.dumps(obj, **kwargs)
Feb 09 17:55:34 3dprinter octoprint[4761]:   File "/usr/lib/python3.7/json/__init__.py", line 238, in dumps
Feb 09 17:55:34 3dprinter octoprint[4761]:     **kw).encode(obj)
Feb 09 17:55:34 3dprinter octoprint[4761]:   File "/usr/lib/python3.7/json/encoder.py", line 199, in encode
Feb 09 17:55:34 3dprinter octoprint[4761]:     chunks = self.iterencode(o, _one_shot=True)
Feb 09 17:55:34 3dprinter octoprint[4761]:   File "/usr/lib/python3.7/json/encoder.py", line 257, in iterencode
Feb 09 17:55:34 3dprinter octoprint[4761]:     return _iterencode(o, 0)
Feb 09 17:55:34 3dprinter octoprint[4761]:   File "/home/razer/OctoPrint-3c5aa60/venv/lib/python3.7/site-packages/OctoPrint-1.4.0.dev2000+g3c5aa60d-py3.7.egg/octoprint/server/util/flask.py", line 1569, in default
Feb 09 17:55:34 3dprinter octoprint[4761]:     return flask.json.JSONEncoder.default(self, obj)
Feb 09 17:55:34 3dprinter octoprint[4761]:   File "/home/razer/OctoPrint-3c5aa60/venv/lib/python3.7/site-packages/Flask-0.12.4-py3.7.egg/flask/json.py", line 81, in default
Feb 09 17:55:34 3dprinter octoprint[4761]:     return _json.JSONEncoder.default(self, o)
Feb 09 17:55:34 3dprinter octoprint[4761]:   File "/usr/lib/python3.7/json/encoder.py", line 179, in default
Feb 09 17:55:34 3dprinter octoprint[4761]:     raise TypeError(f'Object of type {o.__class__.__name__} '
Feb 09 17:55:34 3dprinter octoprint[4761]: TypeError: Object of type bytes is not JSON serializable

So I tried (without success obviously :/)) to understand json encoding process behind permissions and logged dynamic encoders from JsonEncoding Class from utils/json ::

Feb 10 09:22:11 3dprinter octoprint[1297]: OrderedDict([(<class 'frozendict.frozendict'>, <function <lambda> at 0xb2cac5d0>), (<class 'octoprint.access.users.User'>, <function Server.run.<locals>.<lambda> at 0xb2920030>), (<class 'octoprint.access.groups.Group'>, <function Server.run.<locals>.<lambda> at 0xb29200c0>), (<class 'octoprint.access.permissions.OctoPrintPermission'>, <function Server.run.<locals>.<lambda> at 0xb2920108>)])
Feb 10 09:22:11 3dprinter octoprint[1297]: OrderedDict([(<class 'frozendict.frozendict'>, <function <lambda> at 0xb2cac5d0>), (<class 'octoprint.access.users.User'>, <function Server.run.<locals>.<lambda> at 0xb2920030>), (<class 'octoprint.access.groups.Group'>, <function Server.run.<locals>.<lambda> at 0xb29200c0>), (<class 'octoprint.access.permissions.OctoPrintPermission'>, <function Server.run.<locals>.<lambda> at 0xb2920108>)])
Feb 10 09:22:11 3dprinter octoprint[1297]: 2019-02-10 09:22:11,037 - octoprint.plugins.ABL_Expert - INFO - Grid mesh size is 6
Feb 10 09:22:13 3dprinter octoprint[1297]: OrderedDict([(<class 'frozendict.frozendict'>, <function <lambda> at 0xb2cac5d0>), (<class 'octoprint.access.users.User'>, <function Server.run.<locals>.<lambda> at 0xb2920030>), (<class 'octoprint.access.groups.Group'>, <function Server.run.<locals>.<lambda> at 0xb29200c0>), (<class 'octoprint.access.permissions.OctoPrintPermission'>, <function Server.run.<locals>.<lambda> at 0xb2920108>)])
Feb 10 09:22:13 3dprinter octoprint[1297]: OrderedDict([(<class 'frozendict.frozendict'>, <function <lambda> at 0xb2cac5d0>), (<class 'octoprint.access.users.User'>, <function Server.run.<locals>.<lambda> at 0xb2920030>), (<class 'octoprint.access.groups.Group'>, <function Server.run.<locals>.<lambda> at 0xb29200c0>), (<class 'octoprint.access.permissions.OctoPrintPermission'>, <function Server.run.<locals>.<lambda> at 0xb2920108>)])
Feb 10 09:22:13 3dprinter octoprint[1297]: 2019-02-10 09:22:13,174 - octoprint.server.preemptive_cache - INFO - Preemptively caching / (ui _default) for {'base_url': 'https://octo.madby.me/', 'path': '/', 'query_string': 'l10n=en'}
Feb 10 09:22:17 3dprinter octoprint[1297]: OrderedDict([(<class 'frozendict.frozendict'>, <function <lambda> at 0xb2cac5d0>), (<class 'octoprint.access.users.User'>, <function Server.run.<locals>.<lambda> at 0xb2920030>), (<class 'octoprint.access.groups.Group'>, <function Server.run.<locals>.<lambda> at 0xb29200c0>), (<class 'octoprint.access.permissions.OctoPrintPermission'>, <function Server.run.<locals>.<lambda> at 0xb2920108>)])
Feb 10 09:22:17 3dprinter octoprint[1297]: OrderedDict([(<class 'frozendict.frozendict'>, <function <lambda> at 0xb2cac5d0>), (<class 'octoprint.access.users.User'>, <function Server.run.<locals>.<lambda> at 0xb2920030>), (<class 'octoprint.access.groups.Group'>, <function Server.run.<locals>.<lambda> at 0xb29200c0>), (<class 'octoprint.access.permissions.OctoPrintPermission'>, <function Server.run.<locals>.<lambda> at 0xb2920108>)])
Feb 10 09:22:17 3dprinter octoprint[1297]: OrderedDict([(<class 'frozendict.frozendict'>, <function <lambda> at 0xb2cac5d0>), (<class 'octoprint.access.users.User'>, <function Server.run.<locals>.<lambda> at 0xb2920030>), (<class 'octoprint.access.groups.Group'>, <function Server.run.<locals>.<lambda> at 0xb29200c0>), (<class 'octoprint.access.permissions.OctoPrintPermission'>, <function Server.run.<locals>.<lambda> at 0xb2920108>)])
Feb 10 09:22:17 3dprinter octoprint[1297]: OrderedDict([(<class 'frozendict.frozendict'>, <function <lambda> at 0xb2cac5d0>), (<class 'octoprint.access.users.User'>, <function Server.run.<locals>.<lambda> at 0xb2920030>), (<class 'octoprint.access.groups.Group'>, <function Server.run.<locals>.<lambda> at 0xb29200c0>), (<class 'octoprint.access.permissions.OctoPrintPermission'>, <function Server.run.<locals>.<lambda> at 0xb2920108>)])
Feb 10 09:22:17 3dprinter octoprint[1297]: OrderedDict([(<class 'frozendict.frozendict'>, <function <lambda> at 0xb2cac5d0>), (<class 'octoprint.access.users.User'>, <function Server.run.<locals>.<lambda> at 0xb2920030>), (<class 'octoprint.access.groups.Group'>, <function Server.run.<locals>.<lambda> at 0xb29200c0>), (<class 'octoprint.access.permissions.OctoPrintPermission'>, <function Server.run.<locals>.<lambda> at 0xb2920108>)])::ffff:192.168.3.253 - - [10/Feb/2019 09:21:54] "GET /sockjs/info?t=1549786914779 HTTP/1.1" 404 -

I feel something unusual in this list without ability to understand what's going on here

Cheers

[foosel]

@razerraz Did you run into this softwareupdate issue just while checking for updates? I haven't yet encountered it myself, so I wonder what made it pop up on your end.

And thanks for the heads-up regarding razerraz@d269c36

[razerraz]

Well I can't still reproduce it either with all third party plugin disabled. I was guessing having done the same before reporting, right now I doubt about it.
There are still a lot of problems with third parties, mainly import stuff and of course, unicode mess... I've already done PRs for some I'm using, but obviously I miss something.

[foosel]

The plugin situation is going to be a fun one for the foreseeable future. I'm actually pondering not only introducing a flag in the plugin repository but a new control property to signal py3 compatibility during the dual support time.

[smurfix]

Or you could require __plugin_check__ to return something other than True (3, for instance 😄) when called from Python3.

[razerraz]

Maybe the occasion to think about something like a minimal test tool for plugins developers. When I ran into troubles developing one, this idea comes already to my mind...

[foosel]

What are you envisioning there?

[razerraz]

Nothing right now, since I'm quite busy trying to resolve the issue I mention here, and sadly comes from one of my plugin.
I can't understand why my own updater script doesn't want to work :

def get_update_information(self):
        return dict(ABL_Expert=dict(
            displayName='ABL Expert Plugin',
            displayVersion=self._plugin_version,
            current=self._plugin_version,
            type='python_checker',
            python_checker=ABL_VersionCheck,
            pip=('https://framagit.org/razer/Octoprint_ABL_Expert/-/archive' +
                 '/master/Octoprint_ABL_Expert-master.zip')))

class ABL_VersionCheck:
    @classmethod
    def get_latest(cls, target, check, full_data=False, online=True):
        current_version = check.get("current", "1.0.0")
        remote_version = current_version
        if online:
            r = requests.get('https://framagit.org/razer/Octoprint_ABL_Expert'
                             + '/raw/master/VERSION?inline=false')
            if r.status_code == 200:
                remote_version = r.text.strip().encode('ascii', 'ignore')

        info = dict(local=dict(name=current_version, value=current_version),
                    remote=dict(name=remote_version, value=remote_version))

        return info, remote_version == current_version

Maybe not the right place for that, but this was working before with py2

[foosel]

I'll throw it into a test plugin and see if I can reproduce this. Also just fixed an evil windows-only issue caused by an actual CPython bug, fun times.

[foosel]

The problem is that your plugin returns it's latest version as a bytes object, and the json encoder in Py3 dosn't like that. I've pushed the above commits to allow JSON encoding of bytes (by turning them into utf-8 strings) but I'm not sure if that is really the best approach. Happy to discuss.

[smurfix]

its latest version 😛

I can't think of a better solution either, other than telling people "don't use bytes for version strings dammit" which may or may not be a better solution.

[razerraz]

Yeah I just got it on the way for my son's art class, something wrong with the dep in the middle : request...
Just got time to solve the issue before reading you
thanks

[razerraz]

I'll throw it into a test plugin and see if I can reproduce this. Also just fixed an evil windows-only issue caused by an actual CPython bug, fun times.

by the way, what a funny idea some have to use windows for managing a 3dprinter. Some even use it on a raspberry pi. What a pity...

[jneilliii]

Yeah, I've been watching this thread waiting for the plugin developer call out. I'm scared honestly. Would love to have some sort of flag to tell me my stuff is going to break.

[foosel]

by the way, what a funny idea some have to use windows for managing a 3dprinter.

The next thing might scar you for life: I develop this under Windows 😅 Otherwise I wouldn't notice such things, most people run it under Linux:

_its_ latest version 😛

I know, I know... typed too fast, didn't notice until now :P

I'm scared honestly. Would love to have some sort of flag to tell me my stuff is going to break.

We are rather thinking about a flag to not have your plugin load unless you have tested it and know it won't break.

I'm in the middle of merging devel onto the comm refactoring branch and turning that Py3 compatible as well. When I'm through with that (or when I desperately need a break from it to keep the growing frustration at bay - my money is on this coming first...) I'll try a first throw at a py compat control property. Overloading __plugin_check__ is something I'd rather avoid and instead introduce an actual flag, since I can evaluate a flag via the AST without even having to import the plugin. The earlier we can detect incompatibility, the better.

[razerraz]

The next thing might scar you for life: I develop this under Windows sweat_smile Otherwise I wouldn't notice such things, most people run it under Linux:

In fact, it seems to me more natural to use windows for developing than for a web service, even for a pure linux user like me...
And by the way, your graph prove that the windows users are marginal, unlike what I was pointing out...
4 freebsd users, seems like the beginning of a revolution ;)

[foosel]

I've just pushed the above commit (and 8b5626e to fix the broken tests under Py3) which adds a new control property __plugin_pythoncompat__, which is a version compatibility string that gets compared to the detected Python version. If the current version doesn't match, the plugin won't be loaded (and a log line will be generated). It defaults to >=2.7,<3, so support for Python 2 but not 3. As you can see by the need to fix the tests that property is working fairly well as it killed the plugin load tests which run on test plugins that didn't yet set that property 😅

Once a plugin author has checked that their plugin also works under Python 3 they can just explicitly set this property accordingly:

__plugin_pythoncompat__ = ">=2.7,<4"

Once 1.4.0 is out we'll probably start to see some Python 3 only plugins, which can then be marked up accordingly:

__plugin_pythoncompat__ = ">3,<4"

That should cover a lot of bases I hope.

I'll add a similar attribute to the data stored on the plugin repository.

[foosel]

6592be6 also adds evaluation of a new python compatibility flag on the plugin repository (see also https://github.com/OctoPrint/plugins.octoprint.org/commit/ac515fb9d754ef87448359ea37194ed13623bc69), so plugins that don't explicitly state that they are Py3 compatible will show as incompatible in the repository browser.

[razerraz]

fair enough imho

[foosel]

FYI, I've switched the test suite over to pytest since that allows some fancy ALLOW_UNICODE flag on doctests which gets rid of the py2vs3 issues there. It also has a plugin that allows to replace the representation string with something self made which I've now used to produce identical results of e.g. sets but also internal representations (on a different branch), without having to do any convoluted setups in the doc tests.

See commits dcc1148 and 9f2821f.

[jneilliii]

@foosel, let us know the plugin repository changes necessary once you've made that change. I've verified and tested my first plugin and go figure...it's compatible. Of course it is, there isn't hardly any python in it, all client side stuff. Thanks to all the team for this hard work in moving the project forward.

https://github.com/jneilliii/OctoPrint-DragonOrder

[foosel]

I plan to update the repository docs, template and possibly also put out a blog post when things are a bit more settled down, but it will basically be the aforementioned __plugin_pythoncompat__ control property and

compatibility:
  python: ">2.7,<4"

in the front matter of the plugin registration entry .md file.

[jneilliii]

Thanks @foosel. Working on some of my other plugins using the virtual printer has caused some errors. Not sure if it's the plugin's fault or the virtual printer's code. If you'd like me to file a separate issue please let me know. In the meantime, here's one of the errors from the log.

    sleep_match = VirtualPrinter.sleep_regex.match(data)
TypeError: cannot use a bytes pattern on a string-like object
2019-02-22 10:00:32,074 - octoprint.plugins.virtual_printer.VirtualPrinter - ERROR - While handling 'send -0.010\t-0.077\t-0.073\t0.155\t-0.006\t-0.133\t0.110\t0.046\t0.109\t0.173'
Traceback (most recent call last):
  File "c:\users\jneill\documents\octoprint\src\octoprint\plugins\virtual_printer\virtual.py", line 820, in _debugTrigger

[foosel]

Looks like an issue still in the virtual printer - how can I reproduce this?

[jneilliii]

Not sure if it will work the same without the plugin, but try printing this file. It's probably the tabs?

https://github.com/jneilliii/OctoPrint-BedLevelVisualizer/blob/master/virtual_level_report_cartesian.gcode

[foosel]

More a handling of the DEBUG commands in general. Can you paste the full stack trace?

edit never mind, got it. And it's really tiny ^^ This is indeed a general issue with the DEBUG commands, fixing it.

edit 2 should be fixed in 54c8923

[jneilliii]

Thanks @foosel, I can confirm that this commit did resolve my issue and my plugin is now working again with virtual printer...now on to the next plugin...why did I make so many?!?!?!?

[jneilliii]

What would be the reason that the compatibility flag would not be seen by octoprint on startup?

Related repo: https://github.com/jneilliii/octoprint-webcamstreamer/tree/py3/

octoprint.log

[foosel]

@jneilliii I just tried it myself, debugged into things, saw that it ran into an Exception while trying to parse your plugin's source, added some logging for that (see also aacd59a) and now have this in my log for your plugin:

2019-02-27 13:41:57,025 - octoprint.plugin.core - ERROR - Error while parsing AST for webcamstreamer
Traceback (most recent call last):
  File "C:\Devel\OctoPrint\OctoPrint\src\octoprint\plugin\core.py", line 554, in _parse_metadata
    root = ast.parse(f.read())
  File "C:\Python37\Lib\ast.py", line 35, in parse
    return compile(source, filename, mode, PyCF_ONLY_AST)
  File "<unknown>", line 140
    except Exception, e:
                    ^
SyntaxError: invalid syntax
2019-02-27 13:42:00,157 - octoprint.plugin.core - ERROR - Error while parsing AST for webcamstreamer
Traceback (most recent call last):
  File "C:\Devel\OctoPrint\OctoPrint\src\octoprint\plugin\core.py", line 554, in _parse_metadata
    root = ast.parse(f.read())
  File "C:\Python37\Lib\ast.py", line 35, in parse
    return compile(source, filename, mode, PyCF_ONLY_AST)
  File "<unknown>", line 140
    except Exception, e:
                    ^
SyntaxError: invalid syntax
2019-02-27 13:42:00,159 - octoprint.plugin.core - WARNING - Plugin OctoPrint-WebcamStreamer (0.1.2) is not compatible to Python 3.7.0 (compatibility string: >=2.7,<3).

In a nutshell: Your plugin's source code is invalid under Python 3 and doesn't even compile, causing the AST parser to full on its face when trying to read your compatibility flag from it. Which means it defaults to the default of >=2.7,<3 and those flags your plugin as incompatible. Which is actually correct, just with a misleading message (which I'll think about if I can somehow log this better).

In you specific case, try adjusting that exception catch clause to be compatible to both versions.

[foosel]

FYI, I've just added a new setting plugins._forcedCompatible with which you can force OctoPrint to ignore Python compatibility information for certain plugins:

plugins:
  _forcedCompatible:
  - myplugin

Might be helpful during development/migration.

[foosel]

Improved logging thanks to 1ed6730

2019-02-27 15:59:13,549 - octoprint.plugin.core - ERROR - Invalid syntax in C:\Devel\OctoPrint\devenv3\lib\site-packages\octoprint_webcamstreamer\__init__.py for plugin webcamstreamer
Traceback (most recent call last):
  File "C:\Devel\OctoPrint\OctoPrint\src\octoprint\plugin\core.py", line 555, in _parse_metadata
    root = ast.parse(f.read(), filename=path)
  File "C:\Python37\Lib\ast.py", line 35, in parse
    return compile(source, filename, mode, PyCF_ONLY_AST)
  File "C:\Devel\OctoPrint\devenv3\lib\site-packages\octoprint_webcamstreamer\__init__.py", line 140
    except Exception, e:
                    ^
SyntaxError: invalid syntax
2019-02-27 15:59:13,551 - octoprint.plugin.core - WARNING - Plugin OctoPrint-WebcamStreamer (0.1.2) can't be compiled under Python 3.7.0 due to invalid syntax

That should make things more clear I hope.

[DBrianKimmel]

A users report.

I used a new raspi 3B+ and loaded 2018-11-13-raspbian-stretch-lite on a new SD card.
I built a new Python 3.6 on the system and created a link to it from python.
I created a new user 'octo' and did python3.6 -m venv venv3.6
changed to the venv dir and ran source
cloned foosel/octoprint and switched to devel
pip install .
./run in the Octoprint dir
browsed into the new computer:5000
ran thru the setup wizard
uploaded a gcode file
printed it on a monoprice pro
Still printing the raft but everything seems ok

Good work!
Finished the print successfully

[walchko]

so does this work on python3 like @DBrianKimmel says? If so, then pypi should be updated to identify python3 support. Then end of python2.7 is near (pip tells me 1 Jan 2020)! I just tried download from pip and it complained about not finding a matching distro for octoprint.

[foosel]

PyPI has the current stable versions (1.3.x) which aren't yet Python 3 compatible. The devel branch/1.4.x is. The listing on PyPI is correct. I'm aware of the time frame.

[foosel]

1.4.0 has been released.