Pyinstaller: Python macOS builds run from Terminal but crash on Finder launch

[Merkwurdichliebe]

General issue with running an app from the Finder

#1804 seems to be a more general issue with macOS and Python app bundles, regardless of macOS version, the use of Tkinter (#3820) or even pyinstaller itself. Has there been any progress on this issue lately?

To summarise, Python macOS builds (either from pyinstaller or py2app) run just fine from Terminal but crash when launched from the Finder with a double-click on the app icon.

I've added some debug information below. However, as I've said this issue is not directly related to pyinstaller (although it's discussed in several of its issues pages). This is a larger problem. On my end I didn't even succeed building the app once with pyinstaller (I may be doing something wrong), although py2app packaged it without issues. If you follow some SO posts from the link above, some people suggest either one when the other fails.

This, again, seems to confirm that what I describe is not a pyinstaller issue. Complete details are here. I am merely asking if any progress has been made in the last years. Thank you.

Context

• pyinstaller version output: 4.0
• Python version: 3.8.5
• Platform: macOS Catalina 10.15.6 (only platform tested)
• Tried also after installing latest dev build "4.1.dev0"

On execution, pyinstaller write around 20 "missing module" messages in the "warn-app.txt" file. I don't have a clue as to what they are because I didn't import them anywhere. Aside from importing my own files (2 of them), the only imports I have are:

from collections import Counter  
import yaml  
import os
import tkinter as tk
from tkinter import ttk
from webbrowser import open as webopen

None of them are listed in warn-app.text except for yaml (which, again, is not the issue here, see below):

missing module named _yaml - imported by yaml.cyaml (top-level)

Then pyinstaller displays several dozens of those, apparently not importing Tkinter:

WARNING: failed to copy flags of /System/Library/Frameworks/Tcl.framework/Versions/8.5/Resources/Scripts/msgs/*

It then ends with this:

INFO: Building COLLECT COLLECT-00.toc completed successfully.

[bwoodsend]

Ahem, the issue template...

[Merkwurdichliebe]

_Ahem_, the issue template...

Apologies, please indulge a user posting their first issue on GitHub... Fixed (I hope).

[bwoodsend]

Much better

[bwoodsend]

The launching from Finder vs terminal difference is quite likely to just be because its running in a different working directory. Looking at that StackOverflow thread:

file = os.path.realpath('data/data.yml')
with open(file) as f:

That won't work unless you're running from terminal in the root of your application (which Finder does not do). Any other scenario will break it. It should use a path relative to either __file__ or sys._MEIPASS. To test you've done it right:

• Move your compiled app well away from the original source.
• cd in your terminal to somewhere random.
• Launch your app using its full path.
  If it breaks doing that then you can see the error message.

But we're not sure if that is the issue you're facing.

On my end I didn't even succeed building the app once with pyinstaller

Do you mean you can't even compile any code? Even print("hello")? Or you can't compile your specific code?

[Merkwurdichliebe]

Concerning your first point, I've just packaged my app again using py2app (no errors during compile) and followed your suggestions:

• .app in original "dist" folder : open from Terminal works, double-click doesn't.
• .app moved to Desktop : open from Terminal works, double-click doesn't.
• .app copied to Desktop on a separate machine running El Capitan : open from Terminal works, double-click doesn't.

By "open from Terminal" I mean the following commands (I've just tested both on each example above):

open App.app
./App.app/Contents/MacOS/App

As to your second point, I've removed all references to Tkinter in my code (checked and confirmed) but I still get the same Tcl errors mentioned in my post above. I've create a hello.py file with a print statement and pyinstaller still tries (and fails) to package Tcl. Very weird. It doesn't happen with py2app, as opposed to my original issue which happens with both compilers. This problem is outside the scope of my issue I believe, but it may be related. I don't know.

[bwoodsend]

Did you try Terminal cwd in a random location and launching using full path?

As for tkinter, it can get dragged in quite easily - even if you don't use it. Add the --exclude=tkinter option to force PyInstaller to skip it.

[Merkwurdichliebe]

If I understood your suggestion correctly, I created a a folder named "foo" in my home folder and ran this from Terminal :

open /Users/me/foo/App.app

It launches with no issues. Double-click still doesn't work. It seems that the issue lies somewhere else, or am I wrong?

FWIW, I can confirm that pyinstaller compiles my Tk-stripped application successfully if I use --exclude=tkinter. Unfortunately my code isn't much of an application without Tkinter :)

[bwoodsend]

Unfortunately my code isn't much of an application without Tkinter :)

Oops I interpreted I've removed all references to Tkinter in my code as there are no references to Tkinter in my code. We have quite a collection of macOS + problems no-one has managed to solve owing to none of us developers owning macs...

And your last comment shows that opening from Finder isn't a working directory caused issue (although I've no idea what the cause could be). Does the print("hello") example have this issue too?

[Merkwurdichliebe]

And your last comment shows that opening from Finder isn't a working directory caused issue (although I've no idea what the cause could be). Does the print("hello") example have this issue too?

We really have to separate those two issues :

1. The behaviour of my application packaged with py2app is identical wherever the .app bundle is moved to, even to another machine, and is identical to what other users have been reporting when using pyinstaller. Moving the bundle to another directory does not cause issues, as I've described above after following your suggestions. This is the problem I think is the most pressing, as it's been going on for years with nobody (seemingly) knowing why.
2. This problem is much more specific and much less pressing (to me at least): I can't compile my application with Tkinter using pyinstaller. It's not a big deal at the moment. I can do some testing for you on my machine if you can't have a Mac available, but that's something else entirely. I am perfectly happy with the way py2app works because I don't need to use Xcode to get a one-file bundle, as it does this automatically.

I hope it's a bit clearer now :)

[bwoodsend]

I get you. But my last question still stands. I'd like to know if this crashing from Finder issue depends at all on the code being frozen? That'll help narrow it down. We'll worry about the tkinter problem another time...

[Merkwurdichliebe]

Yes, it _does_ depend on the code being frozen. This is the reason I've added the link above to my SO post (which has a lot of diagnostics information BTW). In my case, removing the following statement from my main py file "solved" the issue (I use quotes because obviously it doesn't solve anything):

with open(file) as f:

However, this post on pyinstaller issue #3820 (also linked above) mentions a solution related to Tkinter. That's what's so weird about all of this. The SO post sums it up pretty well, I believe.

[bwoodsend]

I've read the SO post but that post is using filenames relative to cwd which will break your program - PyInstaller or otherwise. But we've verified that you're not doing that, otherwise you'd be seeing errors when you run in Terminal from alternative directories.

What happens if you used an absolute fixed path. i.e. file = "/full/path/to/data/data.yml"?

[Merkwurdichliebe]

Did this, then recompiled with py2app:

# file = os.path.realpath('data/cards.yml')
file = '/Users/me/Files/Docs/Code/Python/MyApp/data/data.yml'
try:
    with open(file) as f:

Exact same behaviour as before (although I didn't go through all the copying to random folders and another Mac part).

[bwoodsend]

OK, now that it odd. There's a try: in there - did you use except FileNotFoundError or just except?

There is another way to test this - you can embed an interactive console session inside your app by putting in the following code:

import code
code.interact(local=globals())

Put it after the file = ... but before the open(file) call. Build it and run it from Finder. Then try playing with the console. Test os.path.exists(file) and open(file, "rb") and open(file) (and anything else that springs to mind). Hopefully you should see a stack-trace for whatever it is causing this whole problem.

[Merkwurdichliebe]

There's a try: in there - did you use except FileNotFoundError or just except?

Sorry, for brevity I only included the first lines when I quoted it. The full code says:

file = '/Users/me/Files/Docs/Code/Python/MyApp/data/data.yml'
    # ...some more irrelevant stuff

    # Read the data.yml file
    try:
        with open(file) as f:
            data = yaml.load(f, Loader=yaml.FullLoader)
    except FileNotFoundError as e:
        print(f'Missing or damaged data.yml configuration file\n({e})')

Put it after the file = ... but before the open(file) call. Build it and run it from Finder. Then try playing with the console. Test os.path.exists(file) and open(file, "rb") and open(file) (and anything else that springs to mind).

>>> import os
>>> file = '/Users/me/Files/Docs/Code/Python/MyApp/data/data.yml'
>>> os.path.exists(file)
True
>>> open(file, "rb")
<_io.BufferedReader name='/Users/me/Files/Docs/Code/Python/MyApp/data/data.yml'>
>>> open(file)
<_io.TextIOWrapper name='/Users/me/Files/Docs/Code/Python/MyApp/data/data.yml' mode='r' encoding='UTF-8'>
>>> import yaml 
>>> with open(file) as f:
...     data = yaml.load(f, Loader=yaml.FullLoader)
>>> print(data)

# Prints just fine

When I Ctrl-D out of the interpreter my GUI appears and everything is OK. This is not a big surprise, though, since I launched the app from the Terminal. If I try to launch it with a double-click I get the same crash, before the GUI or even the Python console appears (although the code.interactstatement is placed as you suggest, right below the file= assignment and before the try: block).

If I remove the open statement and simply read the data from an inline variable, the app runs fine after compilation, even by double-clicking on its icon.

This is starting to look like a variation on the Double-slit experiment.

[bwoodsend]

This is starting to look like a variation on the Double-slit experiment.

Too right it does. That is bizarre.

One last desperate gambit. If you change except FileNotFoundError as e: to except BaseException as e: does anything change (I doubt it will).

[Merkwurdichliebe]

Indeed, no effect. As I wrote in my SO post, this seems to be a much deeper problem related to Python implementation in macOS. Unrelated to Catalina, Tkinter or anything recent. I'm surprised there are so few mentions of this issue, it's been around along time.

[bwoodsend]

Well I'm stumped in that case. I'll leave this issue open and hope someone else has a brain-wave...

[Merkwurdichliebe]

OK, thanks a lot for taking the time to look into it.

[reritom]

@Merkwurdichliebe Do you have a link to a repo with a minimal working (or failing in this case) example, with the commands you used with pyinstaller to generate the bundle?

[reritom]

I may be off-base because I am looking at this without an example repo. I don't think this problem is PyInstaller related, this is specific to how one goes about accessing paths in MacOS.

Basically when you run PyInstaller, it creates a bundle/app. When you double-click the app, it runs it. When you run the app, it runs it in some sort of container.

In my test I had a simple main.py script that would open a static dummy.json that should be located in the same directory as the main.py. Running using the terminal was fine, and if it printed the directory of my main.py, it was as expected. But if you then package the application and run it through finder, the parent directory was / and main.py didn't exist in that directory. But you'd also note that the dummy.json wasn't there either.

So logically one would want to check that PyInstaller was placing the dummy.json into the bundle. Now by default it wasn't, but then I added the --add-datas "app/dummy.json:." to my command (or you update the main.spec with the datas). At which point the dummy.json was now present in the my.app/Contents/Resources.

But the dummy.json still wasn't present in / when running the application through finder. So one wonders "How do I access the resources of the bundle from my script".

And the solution is simply, we use Appkit. So, install appkit using:
pip install pyobjc

Then in your code do the following instead of using a relative path

from AppKit import NSBundle

# NOT path = os.path.join(os.path.dirname(os.path.abspath(__file__)), "dummy.json")
path = NSBundle.mainBundle().pathForResource_ofType_("dummy", "json")

So now when you run the app by double clicking, it will find the related bundle (created by pyinstaller), and allow you to pick the resource you want from your Contents/Resources.

This will allow it to work from double-clicking, but it will no longer work from the command line using python main.py. The path from NSBundle[..] will be None if you run it from the command line. So you may need to branch and fallback to the non-NSBundle approach if the NSBundle approach returns None.

TLDR; you are playing Apples game, so you need to use their toys, specifically AppKit and NSBundle.mainBundle().pathForResource_ofType_(..........).

@Merkwurdichliebe can you confirm that this works for your case, and if it doesn't can you provide an minimal repo that I an use to investigate further.

[Merkwurdichliebe]

I don't think this problem is PyInstaller related, this is specific to how one goes about accessing paths in MacOS.

Please read my original post above and the exchange with @bwoodsend. I insisted from the outset that this was not a problem with pyinstaller and that it was related to macOS disk access through Python. I am not even using pyinstaller. I only posted the issue here because many pyinstaller users have had it too (cf. links in original post above).

from AppKit import NSBundle

# NOT path = os.path.join(os.path.dirname(os.path.abspath(__file__)), "dummy.json")
path = NSBundle.mainBundle().pathForResource_ofType_("dummy", "json")

I installed PyObjC to test this out. The above code indeed doesn't run from Terminal, but it doesn't run from the Finder, either.

I've created a stripped-down version of my main file in a branch named Debug-Minimal, have a look at epidemic.py (which has the NSBundle call you suggest at line 51).

Thanks a lot for looking at this.

[reritom]

@Merkwurdichliebe if you use the command: pyinstaller epidemic.py --onedir --windowed -y --add-data "data/cards.yml:." and update epidemic.py#51 to file = NSBundle.mainBundle().pathForResource_ofType_("cards", "yml") your code will find the correct path to your file.

For me, after that, the app crashes on the master branch due to tkinter related issues that I haven't figured out the solution to yet.

Can you apply those changes and let me know if you manage to have any better luck?

[reritom]

I'm not yet sure what the proper PyInstaller way is for handling the subsequent issue, but using the changes I mentioned in my previous message, and using the hacky approach from this link https://stackoverflow.com/questions/56092383/how-to-fix-msgcatmc-error-after-running-app-from-pyinstaller-on-macos-mojave to fix the tkinter problem (if you encounter it), I am able to run your master branch application from Finder.

Note: neither pyinstaller epidemic.py --onedir --windowed -y --add-data "data/cards.yml:." --hidden-import _tkinter
nor pyinstaller epidemic.py --onedir --windowed -y --add-data "data/cards.yml:." --add-binary='/System/Library/Frameworks/Tk.framework/Tk':'tk' --add-binary='/System/Library/Frameworks/Tcl.framework/Tcl':'tcl' fixed the tkinter issue for me, only manually copying the tcl8 directory based on the aforementioned stackoverflow link worked.

[bwoodsend]

@Merkwurdichliebe Your yaml file:

file = '/Users/me/Files/Docs/Code/Python/MyApp/data/data.yml'

Am I right in saying it is outside the app? I know that Apple Apps are like folders and you can (sometimes?) treat them as such.

[rokm]

Regarding tkinter issue...

This may depend on installation, but on the test system I have (Catalina with Python 3.7.6 from python.org), _find_tcl_tk() call comes up empty.

It seems to use a special function in attempting to find tcl and tk paths:

def _find_tcl_tk_darwin_frameworks(binaries):
    """
    Get an OS X-specific 2-tuple of the absolute paths of the top-level
    external data directories for both Tcl and Tk, respectively.
    Under OS X, Tcl and Tk are installed as Frameworks requiring special care.
    Returns
    -------
    list
        2-tuple whose first element is the value of `${TCL_LIBRARY}` and whose
        second element is the value of `${TK_LIBRARY}`.
    """
    tcl_root = tk_root = None
    for nm, fnm in binaries:
        if nm == 'Tcl':
            tcl_root = os.path.join(os.path.dirname(fnm), 'Resources/Scripts')
        elif nm == 'Tk':
            tk_root =  os.path.join(os.path.dirname(fnm), 'Resources/Scripts')
    return tcl_root, tk_root

But those two checks are not applicable (anymore?), because binaries are:
[('libtcl8.6.dylib', '/Library/Frameworks/Python.framework/Versions/3.7/lib/libtcl8.6.dylib'), ('libtk8.6.dylib', '/Library/Frameworks/Python.framework/Versions/3.7/lib/libtk8.6.dylib')]. And the root path construction with Resources/Scripts does not seem applicable either, because correct root paths are:

• tcl: /Library/Frameworks/Python.framework/Versions/3.7/lib/tcl8.6
• tk: /Library/Frameworks/Python.framework/Versions/3.7/lib/tk8.6

Forcing the use of _find_tcl_tk_dir() instead of _find_tcl_tk_darwin_frameworks() in _find_tcl_tk() results in correct root paths, and the test example from the linked SO post starts correctly.

[Merkwurdichliebe]

@bwoodsend Apple application bundles are indeed folders with the .app extension. Their contents can be navigated easily both in Finder and in Terminal. I have only added the full path to the file variable as a troubleshooting step, when you asked me to earlier on. It used to be file = 'data/data.yml' and later file = os.path.realpath('data/data.yml').

@rokm I think we should open another issue for the Tk problem because it seems unrelated, although I can't be sure. The problem of Python file access under macOS hasn't been solved by removing all the references to Tkinter in my code. Please see this comment above.

@reritom I have never had much luck with pyinstaller, and py2app seems to behave much better with my system (except for this issue that both share, it seems). But I've followed your steps to the letter and, unless I've made a mistake, I come up with an application that doesn't even start, much less crash. After compiling with the options you specified (either followed by the Tk hack or not), a double-click on the app icon doesn't have any visible effect.

I've put two log files in the debug folder of the Debug-Minimal branch. The file called launch-terminal shows the output in Terminal after running pyinstaller with, at the bottom, the commands entered for the Tk hack. The file called launch-console is a colour-coded RTF log from the Console taken immediately following the double-click on the compiled app icon. I hope this helps.

[reritom]

@Merkwurdichliebe Your Debug-Minimal branch still needs line 51 changing from file = NSBundle.mainBundle().pathForResource_ofType_("data/cards", "yml") to file = NSBundle.mainBundle().pathForResource_ofType_("cards", "yml"). At which point, when the bundle is created, I get no errors (based on wrapping the execution in a try/except and writing any errors to a log file).

For your master branch you need the same, and you might need to use the same approach when getting the path for the icon in epidemictk.py#65.

For me, your debug-minimal branch shouldn't do anything if it is working properly. There is no UI and no event loop, so it should just exit immediately, which is what is happening for me once you change line 51.

[bwoodsend]

@Merkwurdichliebe Can you try with a file that isn't inside your app? If that changes anything then it will confirm that @reritom is on the right track.

And we have multiple issues open already about the macOS tkinter problem. We don't need to open a new one. @Merkwurdichliebe, @reritom consider tkinter to be irrelevant to this issue.

[Merkwurdichliebe]

The correct NSBundle code was already in the code when I tested it, I just forgot to update the Debug-Minimal branch. Sorry about that.

Anyway, I just tested again after adding a print and input statement right after the file assignment in the hope of seeing some activity in Terminal, like so:

file = NSBundle.mainBundle().pathForResource_ofType_("cards", "yml")
syslog.syslog(syslog.LOG_ALERT, f'file is {file}')
print(f'file is {file}')

(Note: although there are no references to Tk anywhere in the code, pyinstaller still complains about it missing so I copied the Tk files manually as per the hack.)

The following build command was executed on the Debug-Minimal branch:

pyinstaller epidemic.py --onedir --windowed -y --add-data "data/cards.yml:."

• Running from PyCharm: file is None (as expected)
• Running by double-clicking the epidemic executable inside the epidemic folder created by pyinstaller in the dist folder:

% /Users/me/Files/Docs/Code/Python/Epidemic/pyinstaller/dist/epidemic/epidemic ; exit;
file is /Users/me/Files/Docs/Code/Python/Epidemic/pyinstaller/dist/epidemic/cards.yml
press any key

• Running by double-clicking the .app file built by pyinstaller: nothing
• Running by showing the package contents of the above .app and double-clicking on epidemic.app/Contents/MacOS/epidemic: same as bullet #2 above (file is assigned properly). This even opens the Tkinter interface if I switch to the branch that's using it.

I didn't see a change in behaviour when the file was outside the application bundle.

As I see it, the NSBundle call works when launched from the Finder, as @reritom explains. However, I still can't create a single-file .app executable that works. Asking non-programmers to open a package and dig into the executable before double-clicking it is even more convoluted than asking them to open the app in Terminal (which, again, works in all cases). It seems to be my original issue all over again, on my machine at least: run the file by calling its executable directly works (.app/Contents/MacOS/app) while double-clicking the bundle doesn't.

Also, it might be interesting to note that the NSBundle call doesn't change how py2app behaves, it still crashes on launch even with the Debug-Minimal code.

I should also add that the first sign of unhappiness by macOS, when launching the py2app build with a double-click, is triggered by the TCC system complaining about attribution chains, which apparently has something to do with file access protection:

21:43:11.141461 com.apple.TCC 6636475 198 AttributionChain: REQ:{ID: <ID of InvalidCode>, PID[79890], auid: 501, euid: 501, binary path: '/Users/me/Files/Docs/Code/Python/Epidemic/dist/Epidemic.app/Contents/MacOS/Epidemic'} 

(This is the same Debug-Minimal code used with pyinstaller)

[reritom]

@Merkwurdichliebe when you say Running by double-clicking the .app file built by pyinstaller: nothing, can you elaborate? Does your python application get executed at all? If yes, does it get executed but causes an exception? In the later case you could do something like this to read the error more clearly:

def main():
    try:
        decks = initialize()
        app = App(decks)
    except Exception as e:
        with open("/tmp/epidebug.log", 'w') as f:
            f.write(f"Error {e}")
    else:
        with open("/tmp/epidebug.log", 'w') as f:
            f.write(f"No errors")

Regarding attribution chains, I can't say much, nor can I reproduce your issues. If its a permission based problem maybe you could try:

• Move the epidemic.app into Applications
• Go to System Preferences > Security and Privacy
• Scroll to Full Disk Access and add epidemic.

[Merkwurdichliebe]

@Merkwurdichliebe when you say Running by double-clicking the .app file built by pyinstaller: nothing, can you elaborate? Does your python application get executed at all?

When you double-click a Finder item on macOS, it show a brief animation where the icon is enlarged while losing in opacity. This basically serves as a double-click confirmation from the GUI, alerting the user that the app (or document) is being opened. Then the app is launched. In my case I only get the animation but not much else. The effect is like having a GUI button flash when clicked but not do anything.

• Move the epidemic.app into Applications
• Go to System Preferences > Security and Privacy
• Scroll to Full Disk Access and add epidemic.

Been there, done that, several times over :)

[reritom]

@Merkwurdichliebe and with regard to wrapping the execution to confirm whether the script is being run at all?

[bwoodsend]

@reritom We have tried that already. First with a FileNotFoundError then generalised to BaseException. Neither did anything.

In my case I only get the animation but not much else. The effect is like having a GUI button flash when clicked but not do anything.

@Merkwurdichliebe Can I ask, what actually is your app supposed to do? Is it a GUI application?

[reritom]

@bwoodsend perhaps I have misunderstood, but I read that you suggested wrapping the file opening with the try/except. Using the NSBundle, we get the correct path at least, and @Merkwurdichliebe is able to run the application directly through the executable now. I am now suggesting wrapping the entire script in a try/except to a) see if there are any new errors that we are missing that aren't clearly visible in the system logs due to the executable being executed in a different manner, and b) see if the script is being executed at all once triggered using Finder.

If the script is being executed but has another error, we continue debugging in that direction. If the script isn't be executed at all, we start looking at the stages between clicking the link in Finder, and triggering the execution of the executable. So things like file permissions, paths in the plist, etc.

I am aware that the MacOS GUI is indicating that something is happening with the bouncing animation, but in the background is could be failing before entering the python script, or during execution of the python script based on something we haven't seen yet, hence my questions.

[bwoodsend]

@reritom You're right in that we only wrapped the file operation in the try: except: clause (which we thought was the breaking line). Wrapping the whole thing though would cause the program to skip to the end should it hit an exception.

[Merkwurdichliebe]

@bwoodsend Yes, this is a simple GUI application for tracking statistics on a board game that involves card counting. The master branch (in which there is no trace of PyObj yet) should run on a Windows machine, haven't had time to test it though.

@reritom: Good news, I believe you are getting closer to solving this mystery.

To be clear, from now on I'm using only the executable packaged by pyinstaller in the dist/.app bundle (.app/Contents/MacOS/epidemic), not the executable in the dist/epidemic folder.

Opening the bundle's contents and double-clicking on the executable, as previously, launches the app correctly if the Tk hack has been applied (this is accompanied by a Terminal window warning of a deprecated Tk version, but we agreed to keep Tk out of this...) The log file created in /tmp/ says "No errors".

Double-clicking the bundled .app directly, as previously, has no visible consequence. The log file has this error:

Error 'ascii' codec can't decode byte 0xc3 in position 1202: ordinal not in range(128)%

I don't know if this will lead to a solution, but well done.

EDIT: OMG, it was a French é accented character in one of the string values in the yml file. I can't believe this. It looks like a scene from Brazil. I'm still under shock, will write shortly :)

[bwoodsend]

open(file, encoding="utf-8")

[Merkwurdichliebe]

More test results:

• Removing the é in the file and using the regular os.path.realpath('data/cards.yml') still doesn't work, so @reritom is correct concerning the use of AppKit for file access, regardless of Unicode.
• Removing the é, keeping the NSBundle call and running py2app still causes the same crash.

More questions:

• Will it be possible to read the file using both methods and check for None, or do I need to maintain two py files for separate Window/macOS versions?

I'm still quite confused. I still don't get how all of this was working correctly when launched from Terminal instead of using a double-click on the icon. I will only have time to investigate this further in half a day or so, I'll report back with some more test results as soon I as I have them.

[bwoodsend]

You certainly shouldn't use separate Python files. Worst case scenario should be having to do something like:

import platform

def read_file(filename):
    if platform.system() == "Darwin":
        # macOS only
        from AppKit import NSBundle
        return [Do your fancy NSBundle stuff here]
    else:
        # Otherwise just load the usual way
        with open(filename, "r") as f:
            return f.read()

Then call read_file() whenever you need it.

[bwoodsend]

Is there a read in binary mode NSBundle option equivalent to open(file, "rb")? Then you can just use bytes.decode("utf-8") to choose the encoding.

[reritom]

@Merkwurdichliebe I can't tell from the previous message, but did you have any success using PyInstaller+AppKit+[removing é / using open(file, encoding='utf-8')]?

For your other question, you could either do something like:

path = NSBundle(...)
path = path or your_relative_path

Or perhaps take some other approach looking at sys.executable or sys.platform. Note the above snippet might obscure any errors related to incorrect bundling of resources, but that may not be a big problem.

@bwoodsend wrote his reply as was writing mine, so I would say a mixture between his read_file and the above snippet could be good. Why? His snippet will make your script work cross-platform, but my part nested in his if platform.system() == "Darwin" would enable the script to be run on macos as either an app or as a python script from the command line.

It works through terminal because terminal runs it sort of as a script, which performs as you would expect. Running it through Finder is running it as an app, which means it gets sandboxed and a lot of things we take for granted then have to be achieved using AppKit (like reading command line args, or your case, reading relative paths). I would like to understand more why in your case running the executable has no error, but running the epidemic.app has the encoding issue. That is weird to me.

@bwoodsend the NSBundle is only used for finding the correct path, so your suggestion to use open(file, 'rb') and bytes.decode('utf-8') can be tried directly. Let us know if you have any luck with that @Merkwurdichliebe.

[reritom]

@Merkwurdichliebe If the above approach doesn't work, I have another potential solution.

Read the file using AppKit too.

Add to your epidemic.py imports:

from Foundation import NSFileManager, NSString, NSData, NSUTF8StringEncoding

Then read the file using:

# data = yaml.load(f, Loader=yaml.FullLoader)
raw_data = NSFileManager.defaultManager().contentsAtPath_(NSString(file)) # NSConcreteData
data = NSData(raw_data) # NSData
data = NSString.alloc().initWithData_encoding_(data, NSUTF8StringEncoding) # NSString
data = yaml.safe_load(data) # Read the yml as if it were a normal string

I'm not sure whether the above has any advantages over basic Python file reading, but it could be worth a shot (the above works for me, but I am unable to recreate your encoding issue in any case).

[Merkwurdichliebe]

I'm still trying to get my head around all of this, but @reritom's solution seems to work. Here's the current state of affairs on my end (this is reflected in the current NSBundle branch):

• A short bash script is running pyinstaller with the proper spec file reference and applies the Tk hack:

rm -R build
rm -R dist
pyinstaller Epidemic.spec
cp -r /Library/Frameworks/Python.framework/Versions/3.8/lib/tcl8.6 dist/Epidemic.app/Contents/MacOS/tcl
cp -r /Library/Frameworks/Python.framework/Versions/3.8/lib/tk8.6 dist/Epidemic.app/Contents/MacOS/tk
cp -r /Library/Frameworks/Python.framework/Versions/3.8/lib/tcl8 dist/Epidemic.app/Contents/MacOS/tcl8

• I've had a hard time figuring out the correct option for the spec file, because (1) I was getting annoyed with the long pyinstall command, (2) I had to import two data files, not one, (3) I wanted to configure an app icon and (4) the Tk window wasn't using hi-res Retina resolution. I ended up with the following .spec file which works (I only copy the relevant sections):

a = Analysis(['epidemic.py'],
             pathex=['/Users/me/Files/Docs/Code/Python/Epidemic'],
             binaries=[],
             datas=[('data/cards.yml', 'data'), ('img/pandemic-logo.png', 'img')],
             hiddenimports=[],
             hookspath=[],
             runtime_hooks=[],
             excludes=[],
             win_no_prefer_redirects=False,
             win_private_assemblies=False,
             cipher=block_cipher,
             noarchive=False)
app = BUNDLE(coll,
             name='Epidemic.app',
             icon='icon.icns',
             bundle_identifier=None,
             info_plist={
                  'NSHighResolutionCapable': 'True'
                })

• There's a function in the main script that follows both ideas by @reritom and @bwoodsend (could be condensed but it's clearer for me to keep it this way):

def get_filename(filename):
    name = os.path.splitext(filename)[0]
    ext = os.path.splitext(filename)[1]
    if platform.system() == "Darwin":
        from AppKit import NSBundle
        file = NSBundle.mainBundle().pathForResource_ofType_(name, ext)
        return file or os.path.realpath(filename)
    else:
        return os.path.realpath(filename)

• The file is opened like this:

    file = get_filename('data/cards.yml')
    # ...
    try:
        with open(file, encoding='utf-8') as f:
            data = yaml.load(f, Loader=yaml.FullLoader)
    except FileNotFoundError as e:
        print(f'Missing or damaged cards.yml configuration file\n({e})')

This works as expected, parses the é correctly and can be run by a double-click on the icon and with a Terminal command. My hat's off to you, sirs.

This doesn't fix the py2app crash, BTW. I will try to figure that one out later when I have some time.

There's still the issue of Tk, because I'm still using a hack in my bash script and because the above method doesn't seem to behave nicely with Tk.PhotoImage(file) — when replacing my normal code (self.img_logo = tk.PhotoImage(file='img/pandemic-logo.png')) with the platform-independent function call above, it doesn't work. Does Tk need a different file object? And should I open a new issue concerning Tk?

Thanks a lot for your time and effort.

[reritom]

@Merkwurdichliebe I can't advise for py2app because I have never used it.

I would suggest changing the name of your function from get_filename to get_path, because you already have the filename and you are passing the filename to the function. Get path is more accurate.

For the self.img_logo = tk.PhotoImage(file='img/pandemic-logo.png') line, it should take the same approach as with the json. Just ensure that once PyInstaller is run, the icon file exists in the resources in the bundle.

I'd do something like:

• Make sure the logo file gets copied to Resources by using --add-data "img/pandemic-logo.png:.". Note that in this form, the directory level gets lost, but I would do this because I haven't checked how file = NSBundle.mainBundle().pathForResource_ofType_(name, ext) works for cases where the filename is partially a path. (I can see in your .spec that you have already considered this)
• Then update the logo line.

logo_path = get_path("pandemic-icon.png")
self.img_logo = tk.PhotoImage(file=logo_path)

I think there is another way to handle this too on the MacOs side because the logo can be considered an an asset, which I think is different to a normal resource, but I won't look into that for now unless the icon issue persists.

[bwoodsend]

There is an issue about PyInstaller + tk + an icon + macOS which we couldn't solve on a nabble thread. You may be on the verge of solving that too...

[reritom]

@Merkwurdichliebe for the icon problem, make sure the icon file used for the Finder icon is in icns format. There are online converters for PNG to ICNS.

[Merkwurdichliebe]

@reritom : the icon is not a problem, I had already created a ICNS file and it was referenced in the .spec file. I've only mentioned it as one of the reasons to use the .spec file, instead of the long terminal command. Also, yes, I've figured out how the .spec file expects additional data folders to be specified and, as far as I can tell, the .pathForResource_ofType_ method handle partial paths as expected (e.g. "data/data.yml").

I am trying to come up with a clear workflow to summarise all of this, but I'm now at my laptop rather than my regular desktop machine, and there's a new error which seems to be with PyObjc (?), needed for the NSBundle call. Here's what I did:

• Clone my working NSBundle repo to my laptop.
• Create a venv in the project's folder.
• Install pyinstaller, pyobjc and pyaml using pip.
• Confirm that the application works when run from Terminal.
• Run my bash script above.
• Double-click on the compiled .app bundle: no effect (and no log file create at tmp although the main try/except code hasn't been removed).
• Double-click the executable inside the MacOS folder of the .app bundle. Instead of the app launching, a Terminal window opens with the following output:

Error loading Python lib '/Users/me/Files/Code/Epidemic/dist/Epidemic.app/Contents/MacOS/Python': dlopen: dlopen(/Users/me/Files/Code/Epidemic/dist/Epidemic.app/Contents/MacOS/Python, 10): no suitable image found.  Did find:
    /Users/me/Files/Code/Epidemic/dist/Epidemic.app/Contents/MacOS/Python: code signature invalid for '/Users/me/Files/Code/Epidemic/dist/Epidemic.app/Contents/MacOS/Python'

BTW — and I'm sorry but Tk keeps creeping back into this thread — pyinstaller did not complain about Tcl/Tk files not being found when building the app, although I'm running the same OS and Python version.

[reritom]

@Merkwurdichliebe the new error might be related to this issue https://github.com/pyinstaller/pyinstaller/issues/5062

[Merkwurdichliebe]

That issue page has link to a forum post where @bwoodsend says pyinstaller is not compatible with Python 3.8 and that builds should be done only with 3.7. Is that the case?

Back to desktop, steps taken:

• Fetched updated NSBundle branch.
• Created a venv in the project's folder (I wasn't using one before on the desktop either).
• Activate the virtual environment.
• Install pyinstaller, pyobjc and pyaml using pip.
• Run the bash script without the Tk hack. No errors from pyinstaller.
• Same effects as above (error loading Python lib and dlopen). App won't launch even from Terminal.
• Deactivate the virtual environment.
• Rebuild the app using pyinstaller, which this times does output the Tk errors.
• Apply the Tk hack.
• Double-click the app. Launches normally.

If I take a few steps back and unzoom a bit, developing simple GUI apps for macOS using Python starts to look like a small nightmare. Should I take a crash course on Swift and UIKit instead of banging my head against the wall, or is it even worse? Sorry for the rant.

EDIT: FWIW, with the venv activated, py2app compiled the app with no errors and the bundle can be launched with a double-click. Outside of the venv the compile finishes with no errors but the app won't launch, with the log file in /tmp/ saying Error No module named 'pkg_resources.py2_warn'%. What a mess this is.

[bwoodsend]

@Merkwurdichliebe That nabble thread was a few months back and 3.8 support has been improving quite a bit since then with the final hurdle just fixed yesterday (see #4311).

And if activating a venv seems to fix a broken app then that app is accessing original resources because it doesn't have them bundled (which it should). Whatever the py2app equivalent of --hidden-import=pkg_resources.py2_warn should fix it.

[bwoodsend]

Should I take a crash course on Swift and UIKit instead of banging my head against the wall, or is it even worse?

MacOS and tkinter have always been an ugly pair - even before PyInstaller got involved. Maybe give wxPython a go if you want to switch.

[Merkwurdichliebe]

Too many options to consider right now but IMHO this issue can be closed. It's not really an issue with pyinstaller although, being so difficult to sneaky to locate, maybe pyinstaller could issue some warnings when parsing code destined to be bundled in an .app and which contains open statements.

[gauthierbuttez]

Hi @reritom and @Merkwurdichliebe ,

Thank you so much for your help.

I made a small test with a simple app which open an image and it works.

So I am now working on the source code of my software to change all the way to access to files. I made this little function which handle OS platform (as I will distribute my software to Windows and Mac OS X):

def LoadFile(p_file):
    p_file_split=str(p_file).split('.')
    p_name=p_file_split[0]
    p_ext=p_file_split[1]
    if platform.system() == 'Windows':
        return os.path.join(os.path.dirname(sys.argv[0]), p_file)
    elif platform.system() == 'Darwin':
        return NSBundle.mainBundle().pathForResource_ofType_(p_name, p_ext)

How can I access a directory using NSBundle.mainBundle().pathForResource_ofType ?
The function ask for extension, but I have some case where there are no extension:

• Like loading a CGI directory for some tiny website:
  handler.cgi_directories=["/cgi"]

• Or restarting MyApp executable file which doesn't have extension:
  os.startfile('MyApp')

Should I simply pass an empty string?

handler.cgi_directories=[NSBundle.mainBundle().pathForResource_ofType_("/cgi","")]
os.startfile(NSBundle.mainBundle().pathForResource_ofType_('MyApp',''))

Thanks in advance for your help.

[Merkwurdichliebe]

I haven't a clue but I suggest you try something like None instead. This SO thread says this, although it relates specifically to Swift:

You can easily use the NSBundle method without passing the extension, just pass nil for extension.

[htgoebel]

PyInstaller-Maintainer here. After reading this ticket, I have no clue, what this issue is about:

• Different behaviour between py2app and PyInstaller?
• tkinter not properly imported on OSX? → there are other tickets for this already - please contribute there
• file not found → see #1892 and https://pyinstaller.readthedocs.io/en/stable/runtime-information.html on how to do without platform specific code
• app crashing when started via launcher?
• anything else?
• all of the above → please split into separate tickets so they can be handled

[bwoodsend]

Ahh yeah, this issue went a bit nuts. The key takeaway is this comment which points out that, in order to access data-files, you have to go through NSBundle or it will work from the console but break if launched via Finder.

I agree though that this issue is chaos. I can't follow the conversation and I took part in it. I think after the Conda hooks PR, I'll nick my dad's mac and create a new issue with minimal example etc then close this one.