Sanic: `AttributeError: Can't pickle local object 'register.<locals>._handler'` sanic 19.12.2; python3.8.2; sanic-cors==0.10.0.b1

[endafarrell]

Hi.

On OSX 10.14.6, python 3.8.1

conda list sanic                                                                                                
# packages in environment at /Users/efarrell/miniconda3/envs/LG-py38:
#
# Name                    Version                   Build  Channel
pytest-sanic              1.1.2                    pypi_0    pypi
sanic                     19.12.2                  py38_0    conda-forge
sanic-cors                0.10.0b1                 pypi_0    pypi
sanic-openapi3e           0.6.2                    pypi_0    pypi
sanic-plugins-framework   0.9.0b1                  pypi_0    pypi

I cannot start an API due to

Traceback (most recent call last):
  File "API/src/main.py", line 613, in <module>
    main(sys.argv)
  File "API/src/main.py", line 195, in main
    app.go_fast(**sanic_run_env)
  File "/Users/efarrell/miniconda3/envs/py38/lib/python3.8/site-packages/sanic/app.py", line 1169, in run
    serve_multiple(server_settings, workers)
  File "/Users/efarrell/miniconda3/envs/py38/lib/python3.8/site-packages/sanic/server.py", line 997, in serve_multiple
    process.start()
  File "/Users/efarrell/miniconda3/envs/py38/lib/python3.8/multiprocessing/process.py", line 121, in start
    self._popen = self._Popen(self)
  File "/Users/efarrell/miniconda3/envs/py38/lib/python3.8/multiprocessing/context.py", line 224, in _Popen
    return _default_context.get_context().Process._Popen(process_obj)
  File "/Users/efarrell/miniconda3/envs/py38/lib/python3.8/multiprocessing/context.py", line 283, in _Popen
    return Popen(process_obj)
  File "/Users/efarrell/miniconda3/envs/py38/lib/python3.8/multiprocessing/popen_spawn_posix.py", line 32, in __init__
    super().__init__(process_obj)
  File "/Users/efarrell/miniconda3/envs/py38/lib/python3.8/multiprocessing/popen_fork.py", line 19, in __init__
    self._launch(process_obj)
  File "/Users/efarrell/miniconda3/envs/py38/lib/python3.8/multiprocessing/popen_spawn_posix.py", line 47, in _launch
    reduction.dump(process_obj, fp)
  File "/Users/efarrell/miniconda3/envs/py38/lib/python3.8/multiprocessing/reduction.py", line 60, in dump
    ForkingPickler(file, protocol).dump(obj)
AttributeError: Can't pickle local object 'register.<locals>._handler'

[Tronic]

For triaging: this is now a problem because Python 3.8 changes the default mode of multiprocessing on MacOS to spawn instead of fork. This requires all parameters passed to Sanic worker processed to be picklable, and now there is an object that isn't.

This could be worked around by forcing fork mode instead of the new default, but there really shouldn't be objects that cannot be pickled there because Windows cannot fork and probably could not spawn any workers then either.

[endafarrell]

Hi @Tronic - many thanks for this. We write code on a mixture of OSX and Linux, and deploy to Linux, so in our particular instance, not being able to fork on Windows is not going to cause a problem.

We _do_ have complex objects being passed to the workers, but I had _though_ that those with connections to databases (which we know cannot be pickled) were created only in after_server_start handlers. Will check. Many thanks.

[ashleysommer]

@Tronic a comment related to this:
around the release of Sanic 18.12 we wanted to get multiprocessing working on Windows, so I put in some effort and made _all_ of Sanic pickle-able.

I thought I added tests to ensure a sanic app could be fully pickled and unpickled without error.

Looks like now some code has been introduced at some point that has a local sub_function, which isn't pickleable, or perhaps its only a problem on Python 3.8, which I think our test suite isn't running properly yet.

[Tronic]

@ashleysommer Would it be viable to move most app initialisation to within workers, instead of pickling it from the main process? Only server socket opening really needs to be in the master process, while app definition and other user code could be run in workers. Getting this done would need quite a bit of refactoring, but barring that, it could be the clean solution to this. Doing so would avoid troublesome pickling and the freeze_support hack of multiprocessing that tries to avoid the re-execution of main function and other such code when child processes are run (but it still re-runs any module level code reached prior to that freeze function call).

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. If this is incorrect, please respond with an update. Thank you for your contributions.

[Tronic]

This still needs to be fixed.

[endafarrell]

@Tronic I have some (hopefully good) news.

Our project was busy for a few months but last week we got around to putting effort back into our py38 upgrade. Somewhere along the lines we had also re-checked all code to ensure that calls to set up connections to external databases (we have three external connection pools to three different kinds of database) were made in @bp.listener("after_server_start") methods in our blueprints. There had been at least one "before_server_start" before.

We were happily surprised to find that due to ensuring we created our connections "after_server_start" we can start the sanic 19.12.2 API just fine on OSX 10.14.6 with py3.8.2 (and sanic-cors 0.10.0.post3 - but I do not believe we did anything special with that).

At _this_ stage I can think of two possible next steps:

1: a note in the docs (and a review of existing examples) to state that connections to databases should be created "after_server_start"
2: a note in the docs that states that "after_server_start" happens after the main thread creates the workers but before allowing them to take traffic. It seems somewhat obvious in hind-sight, but I struggled for a little to grok this.

@Tronic - what's the best way to proceed?

[Tronic]

I don't think that before vs. after server start should make a difference for pickling. Both are done in worker processes, and only await loop.create_server(...) occurs between them. Also, what happens inside those handlers should not affect anything.

According to your traceback, the _handler function defined within sanic.static.register is what cannot be pickled, which makes sense because pickling AFAIK doesn't work for anything defined inside functions. Why this is now working is unknown to me, and more time would be needed to fully investigate. Perhaps different Python or Sanic versions, or perhaps somehow it no longer uses that _handler wrapper.

[endafarrell]

Ah - I clearly still don't understand the inner working yet then.

I can't relate the string AttributeError: Can't pickle local object 'register.<locals>._handler' to our code - at least not directly nor in an obvious manner. Do you happen to have any pointers as to how to chase this particular thing down?

[Tronic]

Are you no longer using app.static (for serving static files)? That would explain it because this appears to be the only place where that is used.

[Tronic]

I believe this could be fixed quite easily by rewriting sanic/static.py to use functools.partial to pass use_modified_since, use_content_range, stream_large_files and content_type to _handler instead of it being a local function.

Pinging anyone who cares: pull requests are welcome!

[endafarrell]

@Tronic Sorry for the delay in getting back to you.

Our code doesn't use app.static directly, but has been (for over a year) and still does, use blueprint.static - that part of the code has not changed.