Pelican: "pelican -lr" fails on Windows

[Squareys]

Hi @justinmayer !

I just installed pelican from the latest commit and found pelican -lr to not be working on Windows. Either works, on the other hand (pelican -l or pelican -r).

I get the following output:
~~~
$ pelican -lrD
DEBUG: Pelican version: 3.7.2.dev0
DEBUG: Python version: 3.6.2
DEBUG: Adding current directory to system path
DEBUG: Temporarily adding PLUGIN_PATHS to system path
DEBUG: Restoring system path
CRITICAL: TypeError: can't pickle generator objects
Traceback (most recent call last):
File "c:usersjonathan haleappdatalocalprogramspythonpython36librunpy. py", line 193, in _run_module_as_main
"__main__", mod_spec)
File "c:usersjonathan haleappdatalocalprogramspythonpython36librunpy. py", line 85, in _run_code
exec(code, run_globals)
File "C:UsersJonathan HaleAppDataLocalProgramsPythonPython36Scriptspe lican.exe__main__.py", line 9, in
File "c:usersjonathan haleappdatalocalprogramspythonpython36libsite-p ackagespelican__init__.py", line 563, in main
p1.start()
File "c:usersjonathan haleappdatalocalprogramspythonpython36libmultip rocessingprocess.py", line 105, in start
self._popen = self._Popen(self)
File "c:usersjonathan haleappdatalocalprogramspythonpython36libmultip rocessingcontext.py", line 223, in _Popen
return _default_context.get_context().Process._Popen(process_obj)
File "c:usersjonathan haleappdatalocalprogramspythonpython36libmultip rocessingcontext.py", line 322, in _Popen
return Popen(process_obj)
File "c:usersjonathan haleappdatalocalprogramspythonpython36libmultip rocessingpopen_spawn_win32.py", line 65, in __init__
reduction.dump(process_obj, to_child)
File "c:usersjonathan haleappdatalocalprogramspythonpython36libmultip rocessingreduction.py", line 60, in dump
ForkingPickler(file, protocol).dump(obj)
TypeError: can't pickle generator objects
Traceback (most recent call last):
File "", line 1, in
File "c:usersjonathan haleappdatalocalprogramspythonpython36libmultip rocessingspawn.py", line 99, in spawn_main
new_handle = reduction.steal_handle(parent_pid, pipe_handle)
File "c:usersjonathan haleappdatalocalprogramspythonpython36libmultip rocessingreduction.py", line 87, in steal_handle
_winapi.DUPLICATE_SAME_ACCESS | _winapi.DUPLICATE_CLOSE_SOURCE)
PermissionError: [WinError 5] Zugriff verweigert

~~~

("Zugriff verweigert" means "Access denied")

This is on a basic pelican-quickstart project, no additional plugins, content or custom theme.

Though this may be important for #2162 .

Cheers, Jonathan.

[justinmayer]

Hi Jonathan. I don't use Windows, so I unfortunately won't be of much assistance. Did you modify the settings file in any way?

[Squareys]

No, this is a untouched quickstart project.

[justinmayer]

The error refers to "pickle", which could be related to content caching, which if memory serves should be disabled by default. You could try explicitly disabling caching, but beyond that I'm out of ideas at the moment.

[Squareys]

Disabling caching with

~
CACHE_CONTENT = False
LOAD_CONTENT_CACHE = False
~

or --ignore-cache both did not seem to help. I will see if I can find something later today.

[Squareys]

Debugging this results in an exception in multiprocessing/spawn.py around the following code:

~~~python
def _check_not_importing_main():
if getattr(process.current_process(), '_inheriting', False):
raise RuntimeError('''
An attempt has been made to start a new process before the
current process has finished its bootstrapping phase.

    This probably means that you are not using fork to start your
    child processes and you have forgotten to use the proper idiom
    in the main module:

        if __name__ == '__main__':
            freeze_support()
            ...

    The "freeze_support()" line can be omitted if the program
    is not going to be frozen to produce an executable.''')

~~~

Any ideas? (Adding that "idiom" did not fully fix the error. Now it raises the error in p1.start() )

[Squareys]

Alright, so it looks like on windows it uses pickle to serialize objects and send them to the child process.
With it the arguments passed with watchers, which contains folder watchers, which are generator objects that are not pickleable, that's why it gives us the

CRITICAL: TypeError: can't pickle generator objects

Now, should this be handled? ~I have no experience with pickle, it looks a bit as if I manually would serialize/deserialize every attribute... for Generator and every subclass? 🤔~

Generator objects are not related to Generator class in generator.py. Instead they are some python specific thing. In this case they probably hold a native Windows resource that can not be ~simply~ trivially serialized/deserialized.

(I'm guessing on linux the serialization is handled properly, or the handle to the file watching "thing" is easily shared between processes.... ?)

[Squareys]

Alright, so my suggestion is to have a autoreload_process function which finds its own watchers and then calls autoreload rather than passing autoreload directly to multiprocess.Process(target=here). That would omit the watchers argument and we would not need to serialize it.

[Squareys]

I sketched this out and it would work, but is rather ugly. Potentially better idea:

~python
if args.autoreload and args.listen:
excqueue = multiprocessing.Queue()
p1 = multiprocessing.Process(target=autoreload, ...)
p2 = multiprocessing.Process(target=listen, ...)
p1.start()
p2.start()
exc = excqueue.get()
p1.terminate()
p2.terminate()
logger.critical(exc)
~

This currently spawns two processes, one for listen, one for autoreload. How about having one of the two be done by the current process?
Or maybe use threads instead? (Or are they not as portable? 🤔 )

For any of both I could create a pullrequest once we made a decision.

[segevfiner]

The real error might be CRITICAL: TypeError: can't pickle generator objects due to watchers containing generator objects.

multiprocessing uses fork on Unix systems, but on Windows it runs a new process and pickles the arguments.

[ghost]

It looks like pickling is the real issue like @segevfiner said. I did some looking and found this workaround, and this response.

The response was written by a CPython dev who recommends refactoring the generator into an iterator if custom __getstate__ and __setstate__ methods can't be used.

I'm new to writing iterators, but looking at file_watcher and folder_watcher, it doesn't look like it would be too difficult to refactor the generators into iterators.

[segevfiner]

Another option is refactoring so that you only start the generators in the subprocess. Passing the subprocess the configuration for how to start them.

This should be quite sinpler to do I think.

[ghost]

That's a far better idea, I'll try that before trying to write iterators.

[ghost]

I couldn't get it to work by starting the generators in the subprocess, so I tried the iterator method. It worked (ish), but it turns out that you can't pickle dict_keys objects either.

Looks like this is gonna be a pain :/

[danieleteti]

Any news on this?

[Squareys]

@danieleteti In a comment above I mentioned I have some changes for something that could work, but it's quite a while ago, so I don't remember any details. I hope I still have those changes lying around in case someone wants me to PR those.

Since back in August the interest in this issue was pretty low I opted for pelican -Dr and a local server in the output/ folder instead. That works just as fine.

[danieleteti]

Yes, currently I'm working in the same way (pelican -Dr and python -m http.server in the output/ folder) but would be nice, expecially for newcomers, to have something that just works instead of a workaround.

[jackdewinter]

Do we have any progress on this?

[jackdewinter]

I don't know if this helps anyone else, but I have 2 scripts that contain the (pelican -Dr) and the (python -m http.server) parts build.bat and server.bat. While the extra command windows are a bit annoying, I find that "start build.bat" and "start server.bat" accomplishes what I need.

[justinmayer]

Hi @jackdewinter. As with most other projects, if there were any updates regarding this issue, presumably they would be posted here already.

To summarize, I don't believe any Pelican maintainers currently use Windows — and perhaps don't even have access to computers running Windows. Ergo, it's up to the community to decide whether a reasonably elegant solution can be devised. If and when that day comes, we would be happy to review said contributions. Until that time, I think some further documentation is warranted, if only to explain that adding both --autoreload and --listen to a single pelican invocation on Windows does not currently behave as the user might expect. I've taken the liberty of doing that, in the most succinct way possible, via: 9fdcb90

[thetanil]

I'm a hugo user, but I prefer python, so I thought I would give this a chance. The fact that this issue is 1.5 years old and not fixed is really kind of surprising. No one has to support windows of course, but i just thought I would mention that the experience on Windows surely is discouraging many potential users. Best wishes, looks like a cool tool from the outside at least...

[danieleteti]

I'm a former pelican user... for it's lack of interest in supporting windows, I'm switching to Hugo...
I would to use pelican because it's python, but it's not possible.

[justinmayer]

@thetanil / @danieleteti: Honestly, folks, I find your comments to be disheartening. At the very least, they are inaccurate and show a lack of understanding as how open-source projects work, and at worst, they exhibit a lack of consideration as to how such comments can hurt the feelings of those who donate, for free, their time and effort to maintain said open-source projects.

Contrary to the casually-flung supposition, as far as I am aware, the experience on Windows is not discouraging potential users in significant numbers. In fact, many Windows users happily use Pelican to publish their sites, every day, without any problems whatsoever. Presumably they do so in the same way that I have personally done for years, which is to regenerate in one terminal pane and serve the output in another. This works beautifully and has been the primary way folks have used Pelican for a very long time.

Then one day someone contributed a pull request that added the ability to serve output from within Pelican. That pull request was reviewed and merged. Yay.

Then @Squareys discovered that combining that feature with the regenerate option on Windows resulted in an error. He not only helpfully reported it, but he spent a considerable amount of time working on a potential solution. I regret that he did not receive timely feedback from anyone on his work. Personally, at the time I had recently relocated and had other priorities to attend to.

To suggest that Pelican maintainers lack interest in supporting Windows is not only false, it’s insulting. If someone submits a pull request that addresses a problem, includes tests, and meets the community’s standards, it is merged. This issue is no different in that respect. Contributions that address this issue are welcome.

In short, please take care with your words. Greater attention to accuracy and empathy in this case would have spared some hurt feelings.

[justinmayer]

Do any Pelican users here on Windows use the live-reload Invoke task? If so, how well does it work for you? If you use Windows and haven't tried it, would you mind giving it a try and reporting the results here?

[thetanil]

@justinmayer I really tried to write that comment in the least "choosy begger" way I could. I work open source projects as well, I know how you feel. I think when we work on these projects we also get blind spots because we know how it works. I will try again this way.

My experience as a new user: Find some recommendation, go to the web site, read some docs, follow the quick-start. It doesn't work. Search it up. Land here.

So from my point of view, as a Windows user I did everything right and I still need to build hacky stuff myself to make it work. That's fine, it's open source, I could fix it myself, I understand that. I just thought maybe it's a blind spot and I should at least mention it since the issue is so old.

[thetanil]

Just to be clear and perhaps more irritating, I am not even suggesting a technical change. If there was just some advice on the install docs

Windows Users:

• Don't use make devserver
• Don't use pelican -lr
• instead do x/y

And I have to say, even after reading through this issue, as a new user I am unsure what x/y should be. I see there are a couple workarounds, but I do not know what is correct or best.

[justinmayer]

Now _those_ are useful and actionable suggestions. I agree that should be made clearer. Let's collectively figure out a way to do that.

Since I personally use Invoke rather than Make, I didn't even realize that pelican -lr was being called when users run make devserver. Windows users who use Make should indeed be warned that task will not behave properly.

As for what Windows users _should_ do, I think the first step is determining whether the aforementioned Invoke livereload task works properly on Windows. Anyone care to try and report back?

[ashovlin]

invoke livereload is working for me with Windows 10, Python 3.7, Pelican 4.2.0, livereload 2.6.1 and the output of a fresh pelican-quickstart.

(I did get tripped up for a bit on md/rst files in content/ not triggering the refresh- #2604. https://github.com/lepture/python-livereload/pull/203 is still open, but there are workarounds in that issue. I just removed the wildcard from my local tasks.py for now since I don't use subdirectories in content/)

[thetanil]

i tried notes:

• no invoke command. not in pelican docs. found it and installed
• no livereload module. not in docs. found it installed.
• then I got a traceback not implemented. i think it's cause my content folder is messed up locally
• ran out of time. this is just fyi

[avaris]

https://docs.getpelican.com/en/stable/publish.html#invoke

[justinmayer]

@Squareys: Could you possibly take a look and see whether you still have the changes you worked on related to this issue? They might come in handy. Thanks, Jonathan!

[justinmayer]

Thanks to tireless efforts by @avaris, this issue has been resolved and will be included in the next Pelican release.