Python-for-android: Enabling SSL for python3.5 using crystax

[emillynge]

Hi

I have just spend a full day solving a problem with accessing HTTPS sites on android. I finally figured out the problem:

Almost all http libraries in python relies on the ssl.py module in the standard library. This module is uses a C-extension _ssl.so that must be "baked in" with the cpython interpreter at compilation.
The crystax android ndk however does _not_ do this by default however, and so reaching any https site is not possible. For the novice programmer it is, in no way obvious why ssl.py cannot be imported, so I would propose that somewhere in the documentation, it is described how to recompile the crystax python interpreter _with_ ssl support from the crystax-ndk.

I would gladly write this, but where would it be appropriate to put it in the docs?

Regards
Emil Lynge

Want to back this issue? Post a bounty on it! We accept bounties via Bountysource.

[inclement]

There is a python3 section somewhere in the docs that explains the need to use crystax. You could add further instructions here.

Do you know how to add openssl, using the crystax build scripts to recompile python but compiling openssl first? If not, I can give some information about this (another user pointed it out to me recently - unless that was you?), but it needs some manual hacking of p4a builds right now.

I think the real problem is that crystax's build process currently doesn't build openssl binaries (see https://groups.google.com/forum/#!topic/crystax-ndk/x2LXh3egRuM), but apparently will do so in the next release. I'd guess they'll also compile python against it at that point, so I've been leaving this problem partly to wait for that, though the openssl recipe will need modifying to load openssl differently in this case (rather than compiling it internally).

[emillynge]

Well their build-target-python script actually does a check to see if openssl has been compiled and builds _ssl.so if found.
So I would guess that the problem is resolved when they start building openssl binaries by default.
I have no idea how far out in the future that is, so I don't whether it is worth putting in the effort to fix it here and now..

I guess that it would be possible to make a openssl-crystax recipe that checks if the library has been compiled in the ndk, and recompiles if not. For the build-target-python to work the openssl binaries _must_ be placed in the ndk, so it's difficult to just tweak the existing openssl recipe.

Maybe it is possible to just tweak the python-crystax recipe to copy over, and recompile if openssl is in the build order... hmmmmm..
I may look into that first.

[mgc8]

Hello, are there any news about this issue since last year? It seems that CystaxNDK had a new release, but still without OpenSSL support. Is there a way we can manually add this to Kivy/P4A? Right now it all works nicely with Python3, but the lack of HTTPS kinda makes any serious app impossible to deploy :-/ I'd be happy to test potential patches/solutions that made this happen. Thank you!

[inclement]

@mgc8 This is currently a priority for me, having mostly got through some of the major other things. I don't know when we will support it though.

You can probably manually add it to p4a by working out the steps that p4a itself needs to take to add openssl support. I think it should work to build openssl with crystax (it has a script for this), then build python with CrystaX (it automatically locates the openssl build).

[mgc8]

@inclement thank you for looking into this, I will then attempt to dig deeper into the build process for CrystaX to see how best to add it. If you have any update on official support or a testing version please let me know.

[FabSchilling]

Hello, is there now a working solution to this?

[littlegreenbean33]

I am stuck with same issue here. Gone all the way around from python 3.5 to 2.7 back to 3.5 in order to be able to use https in urlrequest.

stuck at

HTTPSConnection' is not defined

with requirements = kivy, python3crystax,openssl

[tshirtman]

There is a pull request that allows building ssl with crystax, the code hasn't been merged because it really needs a refactor, but a few projects are using it, so it could help you. https://github.com/kivy/python-for-android/pull/1217

you should be able to use it either by telling buildozer to use that branch, or by copying the recipes in your local_recipes directory (also defined in buildozer.spec).

[littlegreenbean33]

Thanks @tshirtman

My application happily runs with https in the kivy launcher.

Any hints on how ssl has been baked into it ? Is there any possibility I could do the same for an converting my application into an apk with ssl support ? what dependencies and specs are used ?

Many Thanks.

JCG

[tshirtman]

I'm pretty sure the kivy launcher runs python2, which should have no issue with ssl, this issue is specifically about the python3crystax build.

[littlegreenbean33]

I was unlucky with python2 ssl then. After more than a week of trying maybe my environment is so deeply corrupted that is helpless.

I need to try again with the stock VM provided. Though it does not work verbatim out of the box. I had compiling problems as well.

Probably will stay put for a while. I need to build a MVP of my application. It rather be without SSL that no MVP with SSL.....I need a fresh mind to look at this again.

[Davincible]

Also trying really hard to get this to work.
Followed the installation instructions of electrum in the Buildozer VM (although trying to compile my own project), but now I get the error
"/home/kivy/Development/SampleBuild/eight_draw/.buildozer/android/platform/build/build/other_builds/python3crystax-version3.6/armeabi-v7a/python3crystax/Modules/_ssl.c:67:25: fatal error: openssl/rsa.h: No such file or directory"

Using armeabi-v7a with API 19 (default), arm64-v8a not availible with API 19.
Python version 3.6
buildozer.spec link
More extensive log

If anybody could point me in the right direction I'd be really grateful. Not sure if this is the right place to post this though

[littlegreenbean33]

SSL is not supported in python3.

If you want to use SSL, you need to move your apk to python 2.7...... I have done extensive research and tests on this. There are comments somewhere where they say is possible.. I failed finding the evidence. It eventually resumes to Crystax not currently supporting ssl out of the box. There are claims that can be manually compile with SSL. I never found instructions clear enough for me to implement it.

[Davincible]

Rewriting the whole application to 2.7 would be too much work. But thanks for your reply.

Will try from scratch in a new environment to compile the Electrum app, and I suppose also ask on a different issue. Will let you know if I make any progress.

[AndreMiras]

@Davincible yes Electrum has it working and I'm also having it working here https://github.com/AndreMiras/EtherollApp/tree/v20180617/src/python-for-android/recipes/python3crystax using the method described by Electrum.

[Davincible]

@AndreMiras Do you also get the many 'hashtype not supported' warnings upon running the apk through logcat?

[AndreMiras]

Yes I do, here's the extract:

06-26 15:13:33.455  5809  5977 I python  : main.py                                                                                                                                                                                                                              
06-26 15:13:34.449  5809  5977 I python  : ERROR:root:code for hash blake2b was not found.                                                                                                                                                                                      
06-26 15:13:34.449  5809  5977 I python  : Traceback (most recent call last):                                                                                                                                                                                                   
06-26 15:13:34.449  5809  5977 I python  :   File "/data/user/0/com.github.andremiras.etheroll/files/app/crystax_python/stdlib.zip/hashlib.py", line 243, in <module>                                                                                                           
06-26 15:13:34.449  5809  5977 I python  :     globals()[__func_name] = __get_hash(__func_name)                                                                                                                                                                                 
06-26 15:13:34.449  5809  5977 I python  :   File "/data/user/0/com.github.andremiras.etheroll/files/app/crystax_python/stdlib.zip/hashlib.py", line 113, in __get_builtin_constructor                                                                                          
06-26 15:13:34.449  5809  5977 I python  :     raise ValueError('unsupported hash type ' + name)                                                                                                                                                                                
06-26 15:13:34.450  5809  5977 I python  : ValueError: unsupported hash type blake2b                                                                                                                                                                                            
06-26 15:13:34.451  5809  5977 I python  : ERROR:root:code for hash blake2s was not found.                                                                                                                                                                                      
06-26 15:13:34.451  5809  5977 I python  : Traceback (most recent call last):                                                                                                                                                                                                   
06-26 15:13:34.451  5809  5977 I python  :   File "/data/user/0/com.github.andremiras.etheroll/files/app/crystax_python/stdlib.zip/hashlib.py", line 243, in <module>                                                                                                           
06-26 15:13:34.451  5809  5977 I python  :     globals()[__func_name] = __get_hash(__func_name)                                                                                                                                                                                 
06-26 15:13:34.451  5809  5977 I python  :   File "/data/user/0/com.github.andremiras.etheroll/files/app/crystax_python/stdlib.zip/hashlib.py", line 113, in __get_builtin_constructor                                                                                          
06-26 15:13:34.451  5809  5977 I python  :     raise ValueError('unsupported hash type ' + name)                                                                                                                                                                                
06-26 15:13:34.451  5809  5977 I python  : ValueError: unsupported hash type blake2s                                                                                                                                                                                            
06-26 15:13:34.452  5809  5977 I python  : ERROR:root:code for hash sha3_224 was not found.                                                                                                                                                                                     
06-26 15:13:34.452  5809  5977 I python  : Traceback (most recent call last):                                                                                                                                                                                                   
06-26 15:13:34.452  5809  5977 I python  :   File "/data/user/0/com.github.andremiras.etheroll/files/app/crystax_python/stdlib.zip/hashlib.py", line 243, in <module>                                                                                                           
06-26 15:13:34.452  5809  5977 I python  :     globals()[__func_name] = __get_hash(__func_name)                                                                                                                                                                                 
06-26 15:13:34.452  5809  5977 I python  :   File "/data/user/0/com.github.andremiras.etheroll/files/app/crystax_python/stdlib.zip/hashlib.py", line 113, in __get_builtin_constructor                                                                                          06-26 15:13:34.452  5809  5977 I python  :     raise ValueError('unsupported hash type ' + name)                                                                                                                                           
06-26 15:13:34.452  5809  5977 I python  : ValueError: unsupported hash type sha3_224                                                                                                                                                                
06-26 15:13:34.453  5809  5977 I python  : ERROR:root:code for hash sha3_256 was not found.                                                                                                                           
06-26 15:13:34.453  5809  5977 I python  : Traceback (most recent call last):            
06-26 15:13:34.453  5809  5977 I python  :   File "/data/user/0/com.github.andremiras.etheroll/files/app/crystax_python/stdlib.zip/hashlib.py", line 243, in <module>
06-26 15:13:34.453  5809  5977 I python  :     globals()[__func_name] = __get_hash(__func_name)
06-26 15:13:34.453  5809  5977 I python  :   File "/data/user/0/com.github.andremiras.etheroll/files/app/crystax_python/stdlib.zip/hashlib.py", line 113, in __get_builtin_constructor
06-26 15:13:34.453  5809  5977 I python  :     raise ValueError('unsupported hash type ' + name)                                                                                                                                    
06-26 15:13:34.453  5809  5977 I python  : ValueError: unsupported hash type sha3_256                                                                                                                                             
06-26 15:13:34.454  5809  5977 I python  : ERROR:root:code for hash sha3_384 was not found.
06-26 15:13:34.454  5809  5977 I python  : Traceback (most recent call last):     
06-26 15:13:34.454  5809  5977 I python  :   File "/data/user/0/com.github.andremiras.etheroll/files/app/crystax_python/stdlib.zip/hashlib.py", line 243, in <module>                                                                   
06-26 15:13:34.454  5809  5977 I python  :     globals()[__func_name] = __get_hash(__func_name)
06-26 15:13:34.454  5809  5977 I python  :   File "/data/user/0/com.github.andremiras.etheroll/files/app/crystax_python/stdlib.zip/hashlib.py", line 113, in __get_builtin_constructor
06-26 15:13:34.454  5809  5977 I python  :     raise ValueError('unsupported hash type ' + name)
06-26 15:13:34.454  5809  5977 I python  : ValueError: unsupported hash type sha3_384
06-26 15:13:34.456  5809  5977 I python  : ERROR:root:code for hash sha3_512 was not found.                                                                                                                                              
06-26 15:13:34.456  5809  5977 I python  : Traceback (most recent call last):                                                                                                                                                        
06-26 15:13:34.456  5809  5977 I python  :   File "/data/user/0/com.github.andremiras.etheroll/files/app/crystax_python/stdlib.zip/hashlib.py", line 243, in <module>
06-26 15:13:34.456  5809  5977 I python  :     globals()[__func_name] = __get_hash(__func_name)            
06-26 15:13:34.456  5809  5977 I python  :   File "/data/user/0/com.github.andremiras.etheroll/files/app/crystax_python/stdlib.zip/hashlib.py", line 113, in __get_builtin_constructor
06-26 15:13:34.456  5809  5977 I python  :     raise ValueError('unsupported hash type ' + name)
06-26 15:13:34.456  5809  5977 I python  : ValueError: unsupported hash type sha3_512                                                                 
06-26 15:13:34.457  5809  5977 I python  : ERROR:root:code for hash shake_128 was not found.                                                          
06-26 15:13:34.457  5809  5977 I python  : Traceback (most recent call last):             
06-26 15:13:34.457  5809  5977 I python  :   File "/data/user/0/com.github.andremiras.etheroll/files/app/crystax_python/stdlib.zip/hashlib.py", line 243, in <module>
06-26 15:13:34.457  5809  5977 I python  :     globals()[__func_name] = __get_hash(__func_name)   
06-26 15:13:34.457  5809  5977 I python  :   File "/data/user/0/com.github.andremiras.etheroll/files/app/crystax_python/stdlib.zip/hashlib.py", line 113, in __get_builtin_constructor
06-26 15:13:34.457  5809  5977 I python  :     raise ValueError('unsupported hash type ' + name)
06-26 15:13:34.457  5809  5977 I python  : ValueError: unsupported hash type shake_128         
06-26 15:13:34.458  5809  5977 I python  : ERROR:root:code for hash shake_256 was not found.
06-26 15:13:34.458  5809  5977 I python  : Traceback (most recent call last):              
06-26 15:13:34.458  5809  5977 I python  :   File "/data/user/0/com.github.andremiras.etheroll/files/app/crystax_python/stdlib.zip/hashlib.py", line 243, in <module>
06-26 15:13:34.458  5809  5977 I python  :     globals()[__func_name] = __get_hash(__func_name)
06-26 15:13:34.458  5809  5977 I python  :   File "/data/user/0/com.github.andremiras.etheroll/files/app/crystax_python/stdlib.zip/hashlib.py", line 113, in __get_builtin_constructor
06-26 15:13:34.458  5809  5977 I python  :     raise ValueError('unsupported hash type ' + name)
06-26 15:13:34.458  5809  5977 I python  : ValueError: unsupported hash type shake_256

[Davincible]

@AndreMiras Mine look exactly the same, Do you know if these errors are important (to the ssl functionality), or will ssl will work just fine even with these errors?

[inclement]

@Davincible I don't think these errors are important, they seem to arise from an internal import check and probably should be taken as info/warnings. Although not explicitly logged, other hashes do work okay, and I don't think these ones are essential.

[AndreMiras]

Yes I confirm it doesn't cause any trouble to my apps.

[inclement]

Closing as the crystax build is obseleted by the new python3 recipe.

[Davincible]

Wait we don't need to use crystax anymore? That's exciting

[inclement]

Yep, we now support python3.7 using the normal google NDK.