Pyinstaller: Compiling using pyinstaller external pythonpath that is in another program module

[AlexFDias]

I have created a program in python where it call's an external python module from a commercial program. As far i have come in my analyses, I have to mentioned I am not a programmer or an expert, this module is a python over C. The lib is in pyd extension and in order to be call in the python script it has to have a environment variable call pythonpath an only can be used in python 3.4 or 3.5. I manage to circumvent this by usingsys.path.insert(0, path)where path is the path to the module to be load or call without going through environment variable or the need to have it defined. The module structure is as follows:

- caris\
  - __init__.py
  - _py_caris.pyd
- caris\bathy\
- caris\ui\

My script has a PyQt5 gui and runs over or is a "wrapper" over this module. When compiling with pyinstaller the processed starts with a bat file.

"C:\Program Files\Python35\Scripts\pyinstaller.exe" -n "Point Cloud" --clean -y ^
-c --debug=imports ^
--exclude-module=caris ^
--add-data="ReadAPointCloudFromAPLYFileExample_01.png;." ^ ^
--icon=Point_Cloud.ico ^
PC.py

I have tried some variations but no improvement.
This is snip from the output if i do not use--exclude-module=caris when compiling.

133910 WARNING: lib not found: cscataloginfou.dll dependency of C:\Program Files\CARIS\BASE Editor\5.3\python\3.5\caris_py_caris.pyd 133925 WARNING: lib not found: gdalrasteru.dll dependency of C:\Program Files\CARIS\BASE Editor\5.3\python\3.5\caris_py_caris.pyd 133925 WARNING: lib not found: exceptionu.dll dependency of C:\Program Files\CARIS\BASE Editor\5.3\python\3.5\caris_py_caris.pyd 133925 WARNING: lib not found: csbsbu.dll dependency of C:\Program Files\CARIS\BASE Editor\5.3\python\3.5\caris_py_caris.pyd

This is the results when running the exe is (exemplar below is for the command above) with --exclude-module=caris:

import search # PyInstaller PYZ
# caris not found in PYZ
# bytecode is stale for 'caris'
# code object from C:\Program Files\CARIS\BASE Editor\5.3\python\3.5\caris\__init__.py
# caris._py_caris not found in PYZ
Traceback (most recent call last):
  File "PC.py", line 19, in <module>
  File "<frozen importlib._bootstrap>", line 968, in _find_and_load
  File "<frozen importlib._bootstrap>", line 957, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 673, in _load_unlocked
  File "c:\program files\python35\lib\site-packages\PyInstaller\loader\pyimod03_importers.py", line 623, in exec_module
    exec(bytecode, module.__dict__)
  File "search.py", line 7, in <module>
  File "<frozen importlib._bootstrap>", line 968, in _find_and_load
  File "<frozen importlib._bootstrap>", line 957, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 673, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 697, in exec_module
  File "<frozen importlib._bootstrap>", line 222, in _call_with_frames_removed
  File "C:\Program Files\CARIS\BASE Editor\5.3\python\3.5\caris\__init__.py", line 15, in <module>
    from ._py_caris import *
  File "<frozen importlib._bootstrap>", line 968, in _find_and_load
  File "<frozen importlib._bootstrap>", line 957, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 666, in _load_unlocked
  File "<frozen importlib._bootstrap>", line 577, in module_from_spec
  File "<frozen importlib._bootstrap_external>", line 938, in create_module
  File "<frozen importlib._bootstrap>", line 222, in _call_with_frames_removed
ImportError: DLL load failed: Impossível localizar o procedimento especificado.
[9712] Failed to execute script PC

My questions are:

Why PYZ has a problem with caris._py_caris? What I'm doing wrong or is this a bug?
caris._py_caris is the caris._py_caris.pyd cdould this be a problem for Pyinstaller?
Is this a correct why to do it?

Thank you for your help.

[Legorooj]

The C code - the pyd file - requires a DLL. PyInstaller can't find it, so you'll need to find and add it yourself. (--add-binary)

[AlexFDias]

I don't think I was clear over this. When i run without the --exclude-module=caris I get the DLL list but pyinstaller is enable to find. If i use the Dependency Walker 2.2 https://www.dependencywalker.com/ I can see the existence of sub routines or DLL, C based so in order to use (--add-binary) I would need to have them "unpacked" to add. What I did and it is because the installation path for the caris module is dynamic was to create a class to get the path independently from where it is installed. It is also the reason to use sys.path.insert(0, path) so when caris is call in the program is not path dependent from a environment variable ( this is actually what is in the caris manual but i can't go to every computers a set them in) and it work in the python interpreter.
So i run it with --exclude-module=caris so it looks for it in the expected installation path. Has you can see there is nothing in the manual to read about this. I have looked in to the hook and freeze solutions but with no practical achievement.

Thank you

[Legorooj]

Ok... I'm confused now. Can you explain the following point again? Give an example?

• > When i run without the --exclude-module=caris I get the DLL list but pyinstaller is enable to find.

[AlexFDias]

Yes that is correct, I have wrote and this is and I quote

This is snip from the output if i do not use --exclude-module=caris when compiling.

The output I get is this, and is during the pyinstalller compiling process, more clearly when i do not use --exclude-module=caris in the pyinstaller command options, so letting pyinstaller look for the caris module.

133910 WARNING: lib not found: cscataloginfou.dll dependency of C:\Program Files\CARIS\BASE Editor\5.3\python\3.5\caris_py_caris.pyd
133925 WARNING: lib not found: gdalrasteru.dll dependency of C:\Program Files\CARIS\BASE Editor\5.3\python\3.5\caris_py_caris.pyd
133925 WARNING: lib not found: exceptionu.dll dependency of C:\Program Files\CARIS\BASE Editor\5.3\python\3.5\caris_py_caris.pyd
133925 WARNING: lib not found: csbsbu.dll dependency of C:\Program Files\CARIS\BASE Editor\5.3\python\3.5\caris_py_caris.pyd

Let me say it again, I'm not a programer or a computer scientist or other, I use python as a means to an end so what I see is that from the _caris_py_caris.pyd_ and using Dependency Walker 2.2 I believe the file has or is the DLL's missing.

This makes sense? This is what I come to as a conclusion.

The manual for this caris module only allows for python 3.4 or 3.5 and uses pythonpath environment variable to set the path to it.

Thank you

[Legorooj]

Right. So what I said earlier is correct. You need to find all of the DLLs that are shown as "not found", and add them when compiling with the --add-binary=example.dll;.. You should be able to use dependency walker to find the DLLs.

[AlexFDias]

It is me now that are confuse. Can you clarify me why when excluding the module and asking to load then from the install dir i get a error? What I'm missing here because now I look like a fowl.
As I wrote i tried with and without --exclude-module=carisand also changed the way i call this module from using the environment variable pythonpath to use sys.path.insert(0, path)where path is the install dir. What is the difference?
Once again thank you

[Legorooj]

@AlexFDias are you planning to send this executable to other users? And does your program acutally use caris?

[AlexFDias]

Yes this is part of a project i have to sent or to deploy to be users that are only interest in the output and not how to get there making the process friendly to all. One button solution.

Yes it uses, this is very important for the way the Point Cloud, actually the name is related with the main point of the logical algoritmo and is 80% based on the caris program / software, calculations and generating of the end files.
This is the reason I'm trying to understand / learn the compiling process.

Thank you.

[Legorooj]

@AlexFDias then if your program is using caris, why are you passing --exclude-module=caris?

[AlexFDias]

Yes, that is because, in my mind, I'm calling a external module that comes with software and since it can be freely chose where to install it will be call from that installation dir. So if it is a external module and is path dependent i thought, sorry if to you sound stupid, the modules should not be included but call from.
In my program the module is call and used but always from where it is installed and as part of the pack or other way.
I'm thinking wrong?
This is why sys.path.insert(0, path) because i can't go and set the environment variable and also it is part of the code itself to find the versions installed.
This is a license / commercial software that my company uses.

[Legorooj]

So you require your users to have caris installed?

[AlexFDias]

Yes exactly. Caris is the name of a commercial software where this program will interact and almost of all contractors uses and deliver to us in the required formats.

[Legorooj]

Right. So then, is there a problem with running the executable, or does it work fine?

[AlexFDias]

I haven't tried it the way you told me --add-binary=example.dll; not in the office do to covid-19 preparing this for tonight when i'm authorized to test in the facilities to do so. At this point and of course using --exclude-module=caris this comes

caris._py_caris not found in PYZ

So i was trying to understand why the

133925 WARNING: lib not found: csbsbu.dll dependency of C:\Program Files\CARIS\BASE
Editor\5.3\python\3.5caris_py_caris.pyd

when i don't use the --exclude-module=caris and also see if i need to change my code.
If i run the pyinstaller without --exclude-module=caris and with --add-binary=example.dll
I shouldn't need to look for the install dir, correct?
I have also to look for those dll that pyinstaller does not find, not sure if I can find then or if they are "hidden" from search or so on and only the module know where they are in the pc.

Thank you for your help

[Legorooj]

@AlexFDias:

If i run the pyinstaller without --exclude-module=caris and with --add-binary=example.dll

If you do that, then you won't need to look for the install directory. In theory, the executable should work anywhere, even without caris installed. So I'd recommend just figuring out how to include caris in the program, becuase that will mean less finding of import directories.

To do this, don't --exclude-module=caris, and, as you've been doing, find the DLLs and add them to the bundle like you've done with your image file. That should all work fine... Unless caris needs a hook. But cross that bridge when you come to it.

[AlexFDias]

@Legorooj

Well no luck. I think i found all the dll and included them as you said and you where write I should had included before. I couldn't find it because they are all over the place and not in one place.
I adjusted the command to include them and also run with --debug=imports do here you have maybe a more to help you see it.
So when I run the exe

# caris not found in PYZ
# bytecode is stale for 'caris'
# code object from C:\Program Files\CARIS\BASE Editor\5.3\python\3.5\caris\__init__.py
# caris._py_caris not found in PYZ
Traceback (most recent call last):
  File "PC.py", line 19, in <module>
  File "<frozen importlib._bootstrap>", line 968, in _find_and_load
  File "<frozen importlib._bootstrap>", line 957, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 673, in _load_unlocked
  File "c:\program files\python35\lib\site-packages\PyInstaller\loader\pyimod03_importers.py", line 623, in exec_module
    exec(bytecode, module.__dict__)
  File "search.py", line 7, in <module>
  File "<frozen importlib._bootstrap>", line 968, in _find_and_load
  File "<frozen importlib._bootstrap>", line 957, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 673, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 697, in exec_module
  File "<frozen importlib._bootstrap>", line 222, in _call_with_frames_removed
  File "C:\Program Files\CARIS\BASE Editor\5.3\python\3.5\caris\__init__.py", line 15, in <module>
    from ._py_caris import *
  File "<frozen importlib._bootstrap>", line 968, in _find_and_load
  File "<frozen importlib._bootstrap>", line 957, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 666, in _load_unlocked
  File "<frozen importlib._bootstrap>", line 577, in module_from_spec
  File "<frozen importlib._bootstrap_external>", line 938, in create_module
  File "<frozen importlib._bootstrap>", line 222, in _call_with_frames_removed
ImportError: DLL load failed: Impossível localizar o procedimento especificado.
[3124] Failed to execute script PC

I do not know what to do now.

Anyway you wrote about hook, to be totally honest I don't know what you mean but I have read the manual and I don't see exactly where or how to use it.
I would like to have the caris module out of the bundle and this is only because it is a commercial software and I don't this we could have it in our distributions. Is possible to have a bundle where it calls the module and runs it from or as a external source?

Thank you

[Legorooj]

@AlexFDias yes, but you need to make sure the DLLs/module are/is discoverable from the python code.

[AlexFDias]

@Legorooj sorry what do you mean with discoverable? If I call the dll in my code?

[Legorooj]

@AlexFDias you'll need to make sure that python can find the DLLs. sys.path manipulation might work, but it might not. I can't say more because I don't know much about caris.

[AlexFDias]

@Legorooj OK. Let me see if I can understand what you just wrote. I have to change my code to look for the DLL in a way that the bundle kan have then, not call the original caris lib. That is done, possibly using sys.path. Is this correct?
If it is i have no idea how to do that, call DLL in the code is possible?
So caris is a software that creates Geographical Information System outcomes and has a pyd based ( sorry for a bad explanation) calling of the DLL as far I can understand. I have use the Dependency Walker 2.2 and that it looked to me. Also these DLL are place in different dir.
If my interpretation is correct this is to much wo fo something that in the end I can't use sind I have to ask caris permission to include this in my software distribution.
Can I have something close to what my script does? Interact with the caris lib when it runs?
Thank you

[Legorooj]

@AlexFDias in most situations, just adding the path of the caris pyd to sys.path before importing it should work. Try that.

[AlexFDias]

@AlexFDias I'm doing that but not to the pyd. As I mentioned above sys.path.insert(0, path) is used and path is the path to the lib structure also above. So in a way I'm doing this correct? Now point to pyd?
Also how hard is to have it to interact with the lin at run time or is not possible?

[Legorooj]

sys.path.insert(0, path) where path is the directory that the caris pyd is in.

Also how hard is to have it to interact with the lin at run time or is not possible?

I'm afraid I don't understand.

[AlexFDias]

I was asking if it is possible and how difficult would be having at run time, when start the exe, to interact with caris module. In a way not to have in my distribution the caris module and DLL but have this when the caris software is install / present.

[Legorooj]

@AlexFDias in theory, simple enough. Just add the correct folder - the pyd just needs to be discoverable from python. That's standard stuff - the normal python import path manipulation will (should?) work just fine.

[AlexFDias]

OK so if I understand you correctly the key is the pyd? If you see the caris dir structure it has the __init__.py and _py_caris.pyd in all sub folders. I call the main, of course caris by import caris and a "sub lib", sorry i just do not know how to call it, from caris using from caris.coverage import * as coverage you realise this is a sub folder where a __init__.py and _py_caris.pyd exists. This also means that I have to point to this pyd? Is pyd the key in your view?
When i use the sys.path.insert(0, path) is so i don't have to specify an environment variable, and since is installation dependent the caris lib using the environment variable required a personalise installation. This so I can tell you this was never thought to be pointing to this pyd actually I didn't know this was so important in my mind was the __init__.py.

[Legorooj]

@AlexFDias sys.path.insert(0, path) with path = '/path/to/caris/folder':

- path
  - to
    - caris
      - folder
        - caris 
          - __init__.py etc

AKA yes you understood properly.

[AlexFDias]

@Legorooj OK I will change the code and get back to you later. First must set this in the code then test.
Stay safe downunder and thank you for your help.

[AlexFDias]

@Legorooj I hope at fallows translate something smart and sorry for the long post.
Well no success, I did as you told. I cal this to define the path:

 path='{}python\\3.5\\'.format(Base[base[-1]])
sys.path.insert(0, path)

to build the path, the Base variable is created from reading windows registry and retrieve the installation path.
After all you explain and using with {}python\\3.5\\caris\\ and out. The error with {}python\\3.5\\caris\\ is:

import search # PyInstaller PYZ
# caris not found in PYZ
Traceback (most recent call last):
  File "PC.py", line 19, in <module>
  File "<frozen importlib._bootstrap>", line 968, in _find_and_load
  File "<frozen importlib._bootstrap>", line 957, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 673, in _load_unlocked
  File "c:\program files\python35\lib\site-packages\PyInstaller\loader\pyimod03_importers.py", line 623, in exec_module
    exec(bytecode, module.__dict__)
  File "search.py", line 7, in <module>
  File "<frozen importlib._bootstrap>", line 968, in _find_and_load
  File "<frozen importlib._bootstrap>", line 955, in _find_and_load_unlocked
ImportError: No module named 'caris'
[7940] Failed to execute script PC

Without caris is the same as above.
So I believe there is something preventing from loading the caris module. I do now if you agree but the lead me to investigate and after some research i found that sys.modules.keys() sys.modules.values() sys.modules.items() sys.path_importer_cache could give some insights, maybe there is others that could give more but I used this from what I got from the research.
From sys.path_importer_cache it loads 3 paths one that i give an 2 more that actually are walk to, see the list.
C:/Program Files/CARIS/BASE Editor/5.3/python/3.5/
C:/Program Files/CARIS/BASE Editor/5.3/python/3.5caris
C:/Program Files/CARIS/BASE Editor/5.3/python/3.5caris\coverage

From sys.modules.items() I present 4 modules as follows:

('caris', )
('caris._py_caris', )
('caris.coverage', )
('caris.coverage._py_caris_coverage', )
From sys.modules.keys() the above is present as you can see.
caris
caris._py_caris
caris.coverage
caris.coverage._py_caris_coverage

Is the loading process should be this way? Is this of any help?
The error, according to me, is occurring because the C:/Program Files/CARIS/BASE Editor/5.3/python/3.5caris and C:/Program Files/CARIS/BASE Editor/5.3/python/3.5caris\coverage are not "there" can I call it hidden. From sys.modules.items() is clear this is not coming, is not retrieved. Is this the difference between python interpreter and the finalised bundle (exe)?
Sorry for the long post, not meant but I would like to give you more than be sure that I have done what you wrote.
Thank you
Alex

[Legorooj]

@AlexFDias hmm. Can you get it working without running as an exe? Straight from source?

[AlexFDias]

@Legorooj If you mean run the script by clicking two time on it or run from a cmd yes, no problem.
I just can't see what is wrong, maybe caris has made safeguards against.

[Legorooj]

@AlexFDias You're definitely running with --exclude-module=caris? Hmm. Try without that and add the caris path with -p PATH.

[AlexFDias]

@Legorooj sadly I have to say I tried and with the same success. Yesterday I used with and without --exclude-module=caris and the -p Path , in the end the same thing. Is like it see is a stand alone, I don't know if understand what I mean.

[Legorooj]

@AlexFDias I'm afriad I've exhausted my resources. You'll have to contact the caris developers and see if they can help. Or StackOverflow might be of some use.

[AlexFDias]

@Legorooj Well just one last question, is possible from a py script create a compile file, do as they did a pyd file or so. Not compile to a stand alone but to have they file so no code can be altered?
I thank you for your time and help. I will do so. Stay safe.

[Legorooj]

I'd look into Cython (https://cython.org).

[AlexFDias]

@AlexFDias I was think the same. Anyway thanks, thank you