Medusa 🚀 - Python3 support / migration

Precious dev time

p0psicles on 26 Aug 2018

Moving to py3 is not allot of work. But supporting py2 and py3 is. Wanna chip in?

p0psicles on 26 Aug 2018

Most code is already Python 2/3 compatible. There are a few libraries left that need to be converted: https://github.com/pymedusa/Medusa/tree/develop/lib

Any help is greatly appreciated :+1:

medariox on 26 Aug 2018

Moving to py3 is not allot of work. But supporting py2 and py3 is.

It's. I did some test and there are a lot of code not working in python3 and a lot of hacks specific for python2. At some point python2 could be totally removed in future releases.

The more painful issue is external dependencies. Why do you copy 3rd party modules in ext package instead of installing the packages with pip?

The external modules in ext package are old versions non compatible with python3. The first step imho is to remove the "ext" packages, install latest version from pip and check if it's working in python2.
I guess most latest versions of all dependencies are already compatible with python2/3. So you can keep launching python2 releases with latest dependencies and the future migration to python3 will be easier.

ngosang on 26 Aug 2018

All external modules in ext are actually up to date and Python 2/3 compatible. You can verify it here: https://pyup.io/repos/github/pymedusa/Medusa/
So is the rest of the application's code in the medusa folder.
Maybe you are referring the lib folder? The plan is to use setup.py once we have all libs cross compatible.

medariox on 27 Aug 2018

I made a lot of changes and I have medusa running with python3. There are still minor issues but it's functional.
I can open a massive PR so you can test it and give feedback, but I don't want to maintain python2 compatibility. What do you think?

All external modules in ext are actually up to date and Python 2/3 compatible.

Yes, but you have the python2 sources in ext & lib folders. I downloaded python3 sources for beautifulsoup4, yaml, httplib2, python-twitter, github, bencode, unrar2 without many problems.

ngosang on 27 Aug 2018

@ngosang we move very fast here, I wouldn't recommend a massive PR as it'll just make more work for us. If you'd like to help maybe start small.

@medariox should we start an issue to track what's needed to get to full python 3 support just as we do with vue?

OmgImAlexis on 27 Aug 2018

I uploaded my changes here => https://github.com/ngosang/Medusa/commits/feature/python3
95% of the features are working

ngosang on 27 Aug 2018

It needs to also run on py2. We cant break it for the userbase which are currently running py2. So if the module has a py3 only version, we need to add both. And select the correct lib.

If we need to monkey patch it, we might need to use something like six.

p0psicles on 27 Aug 2018

The PR is definitely too big.
@OmgImAlexis sounds like a good idea.

medariox on 27 Aug 2018

not just too big, it's focused to run explicitly on python3.

like:

if from_encoding and isinstance(markup, str):
            warnings.warn("You provided Unicode markup but also provided a value for from_encoding. Your from_encoding will be ignored.")               warnings.warn("You provided Unicode markup but also provided a value for from_encoding. Your from_encoding will be ignored.")
            from_encoding = None                from_encoding = None

Will result in unexpected behavior on python2

p0psicles on 27 Aug 2018

Ok, I will make smaller PR and I will keep compatibility with python2.
Take a look #4980
Disclaimer:

I'm testing careful with both python2 and python3 interpreters. With python2 you should expect the same functionally than development brand. Python3 is a long term goal, first versions won't be functional or even won't run.

ngosang on 27 Aug 2018

🎉2

It's almost done, please help me to test => #4982

ngosang on 27 Aug 2018

👍1

Well not be able to review or merge these in a couple of days. Well need to have people available for this.

Havings said that, your prs are very welcome. Well have to see how to get these on safely.

I was thinking about adding the python versions to tox. But only warn for py3

p0psicles on 28 Aug 2018

Just curious, but why is maintaining python2 compatibility important? I understand why a library might want to maintain compatibility, but pymedusa isn't a library. python3 runs everywhere that python2 does.

bobpaul on 3 Nov 2018

Mainly to offer users a seamless transition.
And some distributions, like synology, netgear appliances might not switch that easily to py3.

p0psicles on 3 Nov 2018

👍2

The code that adds Python 3 support is now merged in the develop branch and will be released with the next Medusa release (0.3.0). If anyone wants to help us testing it, please do.

medariox on 23 Dec 2018

Hi, I'm migrating from SC but I cannot even start Medusa under python3 I get the following error:

ERROR:stevedore.extension:Could not load 'addic7ed': bad magic number in 'configparser': b'\x03\xf3\r\n' ERROR:stevedore.extension:Could not load 'legendastv': bad magic number in 'configparser': b'\x03\xf3\r\n' ERROR:stevedore.extension:Could not load 'tvsubtitles': bad magic number in 'configparser': b'\x03\xf3\r\n' ERROR:stevedore.extension:Could not load 'omdb': bad magic number in 'configparser': b'\x03\xf3\r\n' ERROR:stevedore.extension:Could not load 'tvdb': bad magic number in 'configparser': b'\x03\xf3\r\n' Traceback (most recent call last): File "SickBeard.py", line 8, in <module> from medusa.__main__ import main File "/Users/Admin/Apps/Medusa/medusa/__init__.py", line 24, in <module> initialize() File "/Users/Admin/Apps/Medusa/medusa/init/__init__.py", line 38, in initialize _configure_subliminal() File "/Users/Admin/Apps/Medusa/medusa/init/__init__.py", line 192, in _configure_subliminal from subliminal import provider_manager, refiner_manager File "/Users/Admin/Apps/Medusa/ext/subliminal/__init__.py", line 13, in <module> from .cache import region File "/Users/Admin/Apps/Medusa/ext/subliminal/cache.py", line 4, in <module> from dogpile.cache import make_region File "/Users/Admin/Apps/Medusa/ext/dogpile/cache/__init__.py", line 1, in <module> from .region import CacheRegion, register_backend, make_region # noqa File "/Users/Admin/Apps/Medusa/ext/dogpile/cache/region.py", line 3, in <module> from ..util import NameRegistry File "/Users/Admin/Apps/Medusa/ext/dogpile/util/__init__.py", line 1, in <module> from .nameregistry import NameRegistry # noqa File "/Users/Admin/Apps/Medusa/ext/dogpile/util/nameregistry.py", line 1, in <module> from .compat import threading File "/Users/Admin/Apps/Medusa/ext/dogpile/util/compat.py", line 33, in <module> import configparser ImportError: bad magic number in 'configparser': b'\x03\xf3\r\n'

Master Branch Error:
Traceback (most recent call last): File "SickBeard.py", line 8, in <module> from medusa.__main__ import main File "/Users/Admin/Apps/Medusa/medusa/__init__.py", line 24, in <module> initialize() File "/Users/Admin/Apps/Medusa/medusa/init/__init__.py", line 26, in initialize _configure_mimetypes() File "/Users/Admin/Apps/Medusa/medusa/init/__init__.py", line 112, in _configure_mimetypes mimetypes.add_type('text/css', '.css') File "/usr/local/Cellar/python/3.7.2_2/Frameworks/Python.framework/Versions/3.7/lib/python3.7/mimetypes.py", line 340, in add_type init() File "/usr/local/Cellar/python/3.7.2_2/Frameworks/Python.framework/Versions/3.7/lib/python3.7/mimetypes.py", line 354, in init if os.path.isfile(file): File "/Users/Admin/Apps/Medusa/medusa/init/filesystem.py", line 112, in patched_output return handle_output(f(*args, **kwargs)) File "/usr/local/Cellar/python/3.7.2_2/Frameworks/Python.framework/Versions/3.7/lib/python3.7/genericpath.py", line 33, in isfile return stat.S_ISREG(st.st_mode) AttributeError: 'list' object has no attribute 'st_mode

no issues under python 2.

Commit: 918cfe7ce01808499585ffe6c1356fd5ed369f76
Branch: Dev
OS: macOS 10.14.3
Python ver: 3.7.2

I've setup all the python module requirements for pip3 from the requirements list. some kind of issue with dogpile cache? I don't have .core installed as that conflicts...

XxUnkn0wnxX on 21 Feb 2019

Python 3 compatibility is added in the develop branch and will be available with the next release (3.0.0).

medariox on 21 Feb 2019

I did try the develop branch & I got the error in my previous post:

ERROR:stevedore.extension:Could not load 'addic7ed': bad magic number in 'configparser': b'\x03\xf3\r\n' ERROR:stevedore.extension:Could not load 'legendastv': bad magic number in 'configparser': b'\x03\xf3\r\n' ERROR:stevedore.extension:Could not load 'tvsubtitles': bad magic number in 'configparser': b'\x03\xf3\r\n' ERROR:stevedore.extension:Could not load 'omdb': bad magic number in 'configparser': b'\x03\xf3\r\n' ERROR:stevedore.extension:Could not load 'tvdb': bad magic number in 'configparser': b'\x03\xf3\r\n' Traceback (most recent call last): File "SickBeard.py", line 8, in <module> from medusa.__main__ import main File "/Users/Admin/Apps/Medusa/medusa/__init__.py", line 24, in <module> initialize() File "/Users/Admin/Apps/Medusa/medusa/init/__init__.py", line 38, in initialize _configure_subliminal() File "/Users/Admin/Apps/Medusa/medusa/init/__init__.py", line 192, in _configure_subliminal from subliminal import provider_manager, refiner_manager File "/Users/Admin/Apps/Medusa/ext/subliminal/__init__.py", line 13, in <module> from .cache import region File "/Users/Admin/Apps/Medusa/ext/subliminal/cache.py", line 4, in <module> from dogpile.cache import make_region File "/Users/Admin/Apps/Medusa/ext/dogpile/cache/__init__.py", line 1, in <module> from .region import CacheRegion, register_backend, make_region # noqa File "/Users/Admin/Apps/Medusa/ext/dogpile/cache/region.py", line 3, in <module> from ..util import NameRegistry File "/Users/Admin/Apps/Medusa/ext/dogpile/util/__init__.py", line 1, in <module> from .nameregistry import NameRegistry # noqa File "/Users/Admin/Apps/Medusa/ext/dogpile/util/nameregistry.py", line 1, in <module> from .compat import threading File "/Users/Admin/Apps/Medusa/ext/dogpile/util/compat.py", line 33, in <module> import configparser ImportError: bad magic number in 'configparser': b'\x03\xf3\r\n

thought I could help test it but shame it doesnt even want to start...

XxUnkn0wnxX on 21 Feb 2019

Looks like you have subliminal installed separately, which conflicts with the version that Medusa uses (which is Python 3 compatible). You will need to remove the subliminal version you have installed (usually with pip uninstall subliminal) or replace it with a Python 3 compatible version (which should be in their develop branch).

medariox on 21 Feb 2019

Or you need to cleanup all the .pyc files

p0psicles on 21 Feb 2019

hmm odd I swear I uninstalled subliminal recompiled pip modules & it seems to start now thx...
had to get a fresh copy as well git reset --hard wasn't fixing it for some reason...

XxUnkn0wnxX on 21 Feb 2019

Hi,
Switched to develop branch & python3 in order to test memory leak issue described in #6160
Seems to work but I got the following error which I didn't have with master branch:

2019-02-26 05:26:50 ERROR BACKLOG :: [1c234ec] Casting to int failed.
Stack trace:
File "/usr/lib/python3.5/threading.py", line 914, in _bootstrap_inner
self.run()
File "/usr/local/medusa/medusa/scheduler.py", line 93, in run
self.action.run(self.force)
File "/usr/local/medusa/medusa/search/backlog.py", line 215, in run
self.search_backlog()
File "/usr/local/medusa/medusa/search/backlog.py", line 110, in search_backlog
segments = self._get_segments(series_obj, from_date)
File "/usr/local/medusa/medusa/search/backlog.py", line 185, in _get_segments
ep_obj = series_obj.get_episode(episode['season'], episode['episode'])
File "/usr/local/medusa/medusa/tv/series.py", line 844, in get_episode
ep = Episode(self, season, episode)
File "/usr/local/medusa/medusa/tv/episode.py", line 270, in __init__
self._specify_episode(self.season, self.episode)
File "/usr/local/medusa/medusa/tv/episode.py", line 555, in _specify_episode
sql_results = self.load_from_db(season, episode)
Traceback (most recent call last):
File "/usr/local/medusa/medusa/helper/common.py", line 331, in try_int
return int(candidate)
ValueError: invalid literal for int() with base 10: 'NULL'

lklock on 26 Feb 2019

@lklock
Looks like you messed around with your DB. We can't and won't support custom changes.

medariox on 26 Feb 2019

@medariox somewhat related to this question. I setup medusa using developer branch if I where to switch back to master someday would DB have issues? I don't edit db manually but is there like DB versions between master & developer branch? would I need to re-generate my library from scratch again I where to switch to master?

XxUnkn0wnxX on 28 Feb 2019

Once we release the next version you can simply switch to master with your current DB.

medariox on 28 Feb 2019

The release is out now: https://github.com/pymedusa/Medusa/releases
Thanks everyone for the help!

medariox on 13 Mar 2019

Medusa: Python3 support / migration

Most helpful comment

All 28 comments

Related issues