Zeronet: Necessary code changes to support Python3

[HelloZeroNet]

General

• [x] Run 2to3 on every file to automatically apply language changes
• [ ] Add GPL compatible headers (?) to every source code
• [ ] Change % style variable formatting to .format and _("...") style (to also get automatic escaping)

Source code changes

main.py:

• [x] Review ssl/old gevent patches
• [ ] Move configuration dependent actions (stack size/socket patching) to separate function in Config.py

Content/ContentManager.py:

• [ ] Too heavy: Move verification code to separate ContentVerify.py

Crypt/CryptBitcoin.py:

• [x] Remove old style bitcoin signature code (BitcoinECC lib)

Crypt/CryptConnection.py:

• [x] Remove dead ECC code

File/*

• [ ] Move to Peer directory as PeerServer and PeerRequest to make it more concrete

Peer/PeerServer.py

• [x] Move port checking code to separate PeerPortcheck.py

Ui/UiRequest.py

• [ ] Requires major rebase, remove outdated /media handling

Ui/UiWebsocket.py

• [ ] Move actions to UiWebsocketAction.py

Dependencies

• [x] bencode
  
  
  
  • The current library is not available on PIP, possible top3 replacements: bencode, bencode-python3, python-bencode
  
  
  • After running some test:
  
  
  • The current lib is 2x faster on encode, but 2x slower on decode
  
  
  • Py3 is 2x faster on encode and 4x faster on decode using the bencode pip lib
  
  
• [x] BitcoinECC
  
  
  
  • We can remove this, as it's only used for the old (invalid) site addresses.
  
  
• [x] cssvendor
  
  
  
  • Not a third party lib, created to transform special css tags to vendor specific ones.
  
  
  • Should be compatible with python3
  
  
• [x] geventwebsocket
  
  
  
  • Python3 compatible according to docs
  
  
• [x] merkletools
  
  
  
  • Python3 compatible according to docs
  
  
• [x] opensslVerify
  
  
  
  • Not a third party lib, used to interact with openssl so/dll using ctypes.
  
  
  • Probably it's not Python3 compatible as it is.
  
  
  • [x] Need check how hard to translate it to python3, Result: I was able to port it to Python3
  
  
  • [ ] It could use some refactoring: The code is found at pastebin and the quality is not the best.
  
  
  • [ ] Openssl 1.1 compatibility?
  
  
  • Possible replacements:
  
  
  • https://pyopenssl.org/
    
    
    
    
    
    • Pros: Openssl 1.1 compatibility, Robust API
    
    
    
    • Cons: Has not pure-python dependencies (https://cryptography.io/), could be problematic compiling on different platforms
    
    
    
  
  
  • https://www.pycryptodome.org/
    
    
    
    
    
    • Pros: Does not depends on openssl
    
    
    
    • Cons: Self implementation of crypto
    
    
    
  
  
  • https://github.com/cossacklabs/themis
    
    
    
    
    
    • Pros: Modern, multi-platform crypto lib, supports Open/Boring/Libre SSL
    
    
    
    • Cons: Needs to compile, Not widely used
    
    
    
  
  
  • Last year ZeroMux sent me a coincurve/libsecp256k1 based library:
    
    
    
    
    
    • Pros: Does not depends on openssl, According to the first benchmarks it's 2x-10x faster than openssl
    
    
    
    • Cons: It's harder and longer to compile for different platforms, No support for message encryption (required for ZeroMail)
    
    
    
  
  
  • https://github.com/petertodd/python-bitcoinlib
    
    
    
    
    
    • Pros: Using ctypes, no other binary dependency, relatively lightweight
    
    
    
    • Cons: No openssl 1.1 compatibility, According to github issue it may go unmaintained in the future
    
    
    
  
  
  • https://github.com/Simplexum/python-bitcointx
    
    
    
    
    
    • Pros: For of python-bitcoinlib, libsecp256k1 support for signing (according to my benchmarks it's ~70% faster than openssl on win64)
    
    
    
    • Cons: Less popular
    
    
    
  
  
  • PyCrypto: Looks dead
  
  
  • PyElliptic: Looks dead
  
  
  • TODO: Needs benchmarking of current ctypes vs PyOpenSSL vs Cryptodome vs Themis
  
  
• [x] pyasn1
  
  
  
  • Python3 compatible according to docs
  
  
• [x] pybitcointools
  
  
  
  • Combatible with Python3, but no longer maintained. The latest version avaliable under name of bitcoin in pip
  
  
  • Possible replacements:
  
  
  • https://github.com/richardkiss/pycoin
    
    
    
    
    
    • Pro: Pure python
    
    
    
    • Cons: More robust, than pybitcointools (97 vs 12 files)
    
    
    
  
  
• [x] pyelliptic
  
  
  
  • No longer maintained, but compatible with Python3
  
  
  • Possible replacements: see opensslVerify
  
  
• [x] PySocks
  
  
  
  • Not actively maintained, but compatible with Python3
  
  
  • [x] Earlier had problems with the latest versions used with Tor, has to check it again
  
  
• [x] rsa
  
  
  
  • Python3 compatible according to docs
  
  
• [x] subtl
  
  
  
  • Not avaliable on pip
  
  
  • Probably not compatible with Python3
  
  
  • Have not found simple, lightweight replacement, but it's pretty simple, probably adding the function to ZeroNet client itself should not be hard
  
  
• [x] websocket
  
  
  
  • Python3 compatible according to docs
  
  
  • Named websocket-client in pip
  
  

Binary dependencies

• [x] msgpack-python
  
  
  
  • Python3 compatible according to docs
  
  
• [x] gevent
  
  
  
  • Python3 compatible according to docs
  
  

Dev dependencies

• [x] pyfilesystem (renamed to fs)
  
  
  
  • Python3 compatible according to docs
  
  
  • fs module has been removed, changed to watchdog lib
  
  
• [x] werkzeug (debugger)
  
  
  
  • Python3 compatible according to docs
  
  

Deployment

• [ ] Remove third party libs from src/libs
• [ ] Update requirements.txt
• [ ] Updater script for third party libs
• [ ] Windows binary release
• [ ] Linux binary release
• [ ] macOS binary release
• [ ] Update install instructions in documents

Crypto benchmarks

Platform: Win10, Python 3.7.1 64bit

Signing x1000:

• bitcointx + OpenSSL 1.0.2p compiled by indyproject: 3.22s
• bitcointx + secp256k1 from coincurve pip (600k): 3.27s
• bitcointx + secp256k1 extracted from secp256k1_vc120 nupkg (86k): 3.4s
• cryptocurve: 0.081s

Verification x1000:

• bitcointx + OpenSSL 1.0.2p compiled by indyproject: 1.01s
• cryptocurve: 0.141s

Test

• [x] TestConfig.py
• [x] TestConnectionServer.py
• [x] TestContent.py
• [x] TestContentUser.py
• [x] TestCryptBitcoin.py
• [x] TestCryptConnection.py
• [x] TestDb.py
• [x] TestDbQuery.py
• [x] TestDiff.py
• [x] TestEvent.py
• [x] TestFileRequest.py
• [x] TestHelper.py
• [x] TestMsgpack.py
• [x] TestNoparallel.py
• [x] TestPeer.py
• [x] TestRateLimit.py
• [x] TestSafeRe.py
• [x] TestSite.py
• [x] TestSiteDownload.py
• [x] TestSiteStorage.py
• [x] TestTor.py
• [x] TestTranslate.py
• [x] TestUpnpPunch.py
• [x] TestUser.py
• [ ] TestWeb.py

TODO:

• [x] Remove packed libraries
• [x] Support and test for coincurve/libsecp256k1
• [x] Test Tor connection
• [x] Fix benchmark page
• [x] Fix stats page
• [x] Add coincurve support, benchmark
• [x] Add xz compression support, benchmark
• [x] Test AnnounceLocal plugin
• [x] Test AnnounceShare plugin
• [x] Fix AnnounceShare plugin
• [x] Fix UiPassword plugin
• [ ] Fix Multiuser plugin
• [x] Test Bigfile plugin
• [x] Fix Bigfile plugin
• [x] Test ContentFilter plugin
• [ ] Test Cors plugin
• [x] Test CryptMessage plugin
• [x] Fix CryptMessage plugin: Major issue with pyelliptic on py3, probably we need to find another lib
• [x] Test FilePack plugin
• [x] Test Bootstrapper plugin

[DaniellMesquita]

@mkg20001
Android apk release based on it?
Could Kivy also support iOS and other mobile (BlackBerry, WebOS, Firefox OS, Tizen)?

[mkg20001]

@DaniellMesquita depends on how hard it is to get python3 running on android

[DaniellMesquita]

https://kivy.org/doc/stable/faq.html#does-kivy-support-python-3-x
For iOS build, ZeroNet needs to support PY 2.7 for it

[krixano]

@DaniellMesquita That's.... not what that says.

Edit: It says Python 3 is now usable with Kivy (including for android because of python-for-android supporting python 3 - hence the word "also"), but the build tools for iOS require Python 2.

TL;DR: You don't need python 2 to run on iOS with python 3, you just need it for the build tools.

[dgpv]

Please note that in python-bitcointx, the python package has just been renamed from 'bitcoin' to 'bitcointx' (https://github.com/Simplexum/python-bitcointx/issues/7). Also, use_libsecp256k1_for_signing() function will be gone after ,https://github.com/Simplexum/python-bitcointx/pull/14 gets merged, because signing with libsecp256k1 will be the default.

[ofek]

Hello everyone! Quick questions:

• In the benchmark up top coincurve is not the fastest (it appears, correct me if I read it wrong!) and that should not be the case. Could you please show how that test is set up?
• It says, Cons: It's harder and longer to compile for different platforms, No support for message encryption (required for ZeroMail). Can you please explain what you need for message encryption? Also, there are actually pre-compiled wheels for every major platform 🙂

[HelloZeroNet]

@ofek

• If you look it again it is the fastest. There was performance problem using secp256k1 with other library that is using ctypes for function calls, but it's already fixed. Afterall currently looks like we will use python-bitcoinlib for interacting with openssl and coincurve for libsecp256k1 as an optional speedup for verification of the signatures
• For message encryption/decryption we need AES and ECIES functions

[DaniellMesquita]

Is that AES-256?

[HelloZeroNet]

https://github.com/HelloZeroNet/ZeroNet/blob/master/plugins/CryptMessage/CryptMessage.py#L7
aes-256-cbc by default, but we need openssl anyway for encrypted connections, so it's ok to keep it as it is and use coincurve as optional speedup for the most critical part

[ofek]

@HelloZeroNet Ah makes sense, thanks! Could you please ping me on the PR that adds coincurve as an optional speedup? I'm always curious to see how it's used.

[HelloZeroNet]

@ofek Sure, this is the code what we going to use: https://gist.github.com/HelloZeroNet/c82198335b430b51d86f9d6484fd91c2

I got it from a ZeroNet user (@MuxZeroNet) a few months ago. Please let me know if you have suggestions. (We only going to use it for verification as the signing is pretty rare operation)

[DaniellMesquita]

https://github.com/HelloZeroNet/ZeroNet/blob/master/plugins/CryptMessage/CryptMessage.py#L7
aes-256-cbc by default, but we need openssl anyway for encrypted connections, so it's ok to keep it as it is and use coincurve as optional speedup for the most critical part

AES-256 is quantum resistant in medium-therm. But Bitcoin cryptography (that gives life to zites) isn't.

[ofek]

@HelloZeroNet For the key_pair function you should be able to just return secretkey.public_key, secretkey to avoid double curve computation. If you really, really want decoupled objects then keep the return as-is and for the public key use from_valid_secret to bypass bounds checking.

Please let me know when the PR arrives 🙂

[HelloZeroNet]

First Python3 version is ready for testing: https://github.com/HelloZeroNet/ZeroNet/tree/py3

What's left before release:

• Source code updating
• Linux/Win/MacOS distirbutions
• More testing

[HelloZeroNet]

There was various compatibility problems with Python 3.5 (some of them was tricky to reproduce), but I was able to fix it and it should work now.
The sqlite that comes with Python 3.7 (on windows) seems significantly faster, so it's recommended to use that one.
I have set up automatic test for Python 3.5, 3.6, 3.7 compatibility: https://travis-ci.org/HelloZeroNet/ZeroNet

The last Windows XP compatible Python version is 3.4, so I will try to make it work as well

[krixano]

Ok, so PeerMessage now supports Python 3 Version of ZeroNet - in the py3 branch.

[imachug]

@HelloZeroNet Uploading big files doesn't seem to work.

If you upload a file to KxoNetwork (I just uploaded a plugin), I get this error:

[08:20:11] - UiWSGIHandler error: TypeError: 'NoneType' object is not subscriptable in UiServer.py line 41 > pywsgi.py line 924 > pywsgi.py line 908 > helper.py line 304 > Bigfile/BigfilePlugin.py line 101 > PeerHashfield.py line 56

Sounds like the hash is None.

If I upload a file to ZeroUp, I get this:

[08:34:44] - UiWSGIHandler error: UnicodeDecodeError: 'charmap' codec can't decode byte 0x98 in position 813: character maps to <undefined> in UiServer.py line 41 > pywsgi.py line 924 > pywsgi.py line 908 > helper.py line 304 > Bigfile/BigfilePlugin.py line 102 > OptionalManager/OptionalManagerPlugin.py line 72 > Bigfile/BigfilePlugin.py line 370 > Bigfile/BigfilePlugin.py line 458 > cp1251.py line 23

In case you need the files, here they are:

For KxoNetwork
For ZeroUp

[HelloZeroNet]

@imachug Thanks for reporting, working on it!

[HelloZeroNet]

Should be fixed in Rev4048 https://github.com/HelloZeroNet/ZeroNet/commit/87abdb92e9236c794b9de22c91d5d3d32d83e12f

[imachug]

It's actually half-fixed, I get this error:

[07:47:18] - UiWSGIHandler error: AttributeError: 'bytes' object has no attribute 'encode' in UiServer.py line 41 > pywsgi.py line 924 > pywsgi.py line 908 > helper.py line 304 > Bigfile/BigfilePlugin.py line 72

However, the file is uploaded correctly.

[radfish]

@imachug try with PR #1982

[krixano]

@HelloZeroNet I'm getting this error when I startup ZeroNet. I have the Bootstrap plugin enabled. I'm not sure if you've finished making this plugin work with py3 (I didn't see it in TODO list, I don't think):

[23:00:25] PluginManager Plugin Bootstrapper load error: OperationalError: cannot VACUUM from within a transaction in PluginManager.py line 56 > Bootstrapper/__init__.py line 1 > Bootstrapper/BootstrapperPlugin.py line 11 > Bootstrapper/BootstrapperDb.py line 17 > Bootstrapper/BootstrapperDb.py line 36 > Bootstrapper/BootstrapperDb.py line 45 > Db.py line 106 > DbCursor.py line 101

[imachug]

I think I've seen an issue about Bootstrapper but I can't find it now.

[filips123]

@HelloZeroNet Shouldn't this be closed as ZeroNet now supports Python 3?

Also, can you push some changes to master branch to alert all users which are still using Python 2 that it will be unsupported in 2020 and with details how to update to Python 3 version?

[imachug]

@HelloZeroNet Can we close this now?