Sentry-python: Sourcecode not visible when using PyInstaller

[jaymegordo]

Important Details

How are you running Sentry?
Frozen python app created with PyInstaller

On-Premise w/ Docker, version x.y.z

Saas (sentry.io)

Other, version x.y.z, [briefly describe your environment]

Description

When my app is frozen with pyinstaller, the error context lines are not captured.

Added a couple print statements in sentry_sdk/utils.py:

def get_lines_from_file(
    filename,  # type: str
    lineno,  # type: int
    loader=None,  # type: Optional[Any]
    module=None,  # type: Optional[str]
):
    # type: (...) -> Tuple[List[Annotated[str]], Optional[Annotated[str]], List[Annotated[str]]]
    context_lines = 5
    source = None
    if loader is not None and hasattr(loader, "get_source"):
        try:
            source_str = loader.get_source(module)  # type: Optional[str]
        except (ImportError, IOError):
            source_str = None
        if source_str is not None:
            source = source_str.splitlines()

    print(f'filename: {filename}, module: {module}')
    print(f'source is None: {source is None}')
    if source is None:
        try:
            source = linecache.getlines(filename)
            print(f'source: {source}')
        except (OSError, IOError):
            return [], None, []

and at the end of sentry_sdk\client.py at line 216 in _prepare_event, i just printed print(event['exception']) to see the final event exception passed out. (trimmed some extra non imporant stuff off this)

App not frozen (working):

filename: Y:\OneDrive\Python\SMS\smseventlog\gui\gui.py, module: smseventlog.gui.gui
source is None: False

{'filename': 'smseventlog\\gui\\gui.py', 
'abs_path': 'Y:\\OneDrive\\Python\\SMS\\smseventlog\\gui\\gui.py', 
'function': 'test_error', 
'module': 'smseventlog.gui.gui', 
'lineno': 257, 
'pre_context': ['', '    def test_error(self):', '        a = 5', '        b = 6', ''], 
'context_line': '        return a / 0', 
'post_context': ['', '    def create_actions(self):', '        # Menu/shortcuts', '        t = self.active_table_widget', ''], 
'vars': {'self': '<smseventlog.gui.gui.MainWindow object at 0x000002CD9D0DFAF0>', 'a': '5', 'b': '6'}, 
'in_app': True}]}

App frozen (not working):

filename: smseventlog\gui\gui.py, module: smseventlog.gui.gui
source is None: True
source: []

 {'filename': 'smseventlog\\gui\\gui.py', 
'abs_path': 'C:\\Users\\Jayme\\Documents\\smseventlog\\dist\\smseventlog_win\\smseventlog\\gui\\gui.py', 
'function': 'test_error', 
'module': 'smseventlog.gui.gui', 
'lineno': 257,
 'pre_context': [], 
'context_line': None, 
'post_context': [], 
'vars': {'self': '<smseventlog.gui.gui.MainWindow object at 0x0000022638C4A280>', 'a': '5', 'b': '6'}, 
'in_app': True}]}}]}

the difference seems to be that in the non frozen app, the full filename is recognized Y:\OneDrive\Python\SMS\smseventlog\gui\gui.py, while in the frozen app its only recoginzed as smseventlog\gui\gui.py and the module can't be propery read to find the lines specified in the traceback.

Steps to Reproduce

What you expected to happen

Pre and post context should be captured, even in a frozen app.

Possible Solution

[If you have an idea on how this could be solved include that detail here.]

[untitaker]

That's a known issue in the Python SDK. Moving there.

[jaymegordo]

Thanks @untitaker, in the meantime do you know the best way to include extra data say a stacktrace or whatever else by using eg capture_exception(exception, extra_data='extra stuff here' ?

[untitaker]

Please refer to the documentation: https://docs.sentry.io/platforms/python/enriching-error-data/additional-data/#adding-context

[jaymegordo]

Any progress on this one? would be super helpful to have working thx!!! :)

[becurrie]

Seconding @jaymegordo, having a fix/solution for this would be very beneficial :)

[becurrie]

@jaymegordo @untitaker After reviewing the source-code, I don't think this will be very easy to handle... The file itself in a frozen executable is in relation to the directory it's in (see relevant pyinstaller docs) rather than the absolute path (which is required to grab source code lines). Additionally, even if we can find the file and open it for viewing, it's going to be a .pyc file which will not contain the same exact lines as our normal .py files we want to send stack trace information about (I suppose some de-compilation could be performed if we have the file to get at least something similar to our Python code?) I think a proper fix for this would end up being a pretty big change to the source code and how it handles this edge case.

The best I was able to come up with for the time being is to use the before_send hook (docs) in conjunction with the traceback module in Python, I'm able to do something like this:

def before_send(event, hint):
    event["extra"]["exception"] = ["".join(
        traceback.format_exception(*hint["exc_info"])
    )]
    return event

And my sdk init call:

sentry_sdk.init(
    dsn=__SENTRY_DSN__,
    release=__SENTRY_RELEASE__,
    before_send=before_send,
)

This works well enough so that I at least have _some_ information available in the Sentry logs.

_Edit_ The photo above is actually from a normal .py file sentry event exception being captured. Using a frozen executable results in a Sentry event more like this:

@untitaker Is this a appropriate enough solution for the issue? Or is there something else kicking about in Sentry that could handle this? I only started using Sentry the other day, so a lot of it is still new to me!

[untitaker]

I'm surprised to see that there's any information at all reachable from within the Python process. We use our own "formatting function" on exception, I wonder where format_exception gets this extra info from?

[becurrie]

@untitaker Actually, I just realized I took my screenshot from my .py file sentry event, rather than my frozen executable... The information provided in the additional data "exception" looks like this. I'll update my photo above as well.

Frozen executables do display some useful information, notably though, it removes the line that actually caused the exception being sent along in the event.