Pelican: Is it possible write-selected just doesn't work?

[mlissner]

Issue #2601 seems similar, but I can't figure out how to make --write-selected work. I've tried:

↪ pelican content -s pelicanconf.py -d -D --write-selected output/index.html
↪ pelican content -s pelicanconf.py -d -D --write-selected /home/me/programming/website/output/index.html
↪ pelican content -s pelicanconf.py -d -D --write-selected 'output/*'
↪ pelican content -s pelicanconf.py -d -D --write-selected ["output/index.html"]

And a dozen other things. Whenever I try these, they just don't generate any markdown files, though they do copy over a ton of assets. As far as I can tell, it really doesn't work, even with the example that is in the documentation.

I'm on version 4.2. If there's a fix here I should know about, I'd love to hear it. Maybe I'm just using it wrong? But I'm using output/index.html, which seems pretty hard to screw up.

[mlissner]

OK, no, I don't think it ever works, and I think #2601 (opened August 2019, marked stale Oct. 1, 2019) was totally right.

In utils.py there's a function that compares the write-selected setting/command line arg vs the file that's currently being processed. I added a few print statements, but otherwise it looks like this:

89 def is_selected_for_writing(settings, path):
890     '''Check whether path is selected for writing
891     according to the WRITE_SELECTED list
892 
893     If WRITE_SELECTED is an empty list (default),
894     any path is selected for writing.
895     '''
896     if settings['WRITE_SELECTED']:
897         print("Setting is: %s" % settings['WRITE_SELECTED'])
898         print("Path is: %s" % path)
899         if path not in settings['WRITE_SELECTED']:
900             print("Path is not in settings.")
901         return path in settings['WRITE_SELECTED']
902     else:
903         return True

Unfortunately, path generated by Pelican as a relative path and the write-selected value is an absolute path, even if you put a relative one into your command.

So, I use a relative path in my command, like so:

pelican content -s pelicanconf.py --write-selected 'output/legal-api-downloader/index.html'

The tweaked code above yields:

Setting is: [u'/home/mlissner/Programming/free.law/output/legal-api-downloader/index.html']
Path is: output/legal-api-downloader/apple//index.html
Path is not in setting.

Also note that path has two slashes, so I guess this doesn't support the slug field when it ends in a slash either.

The fix, I suppose, would be to either always use relative or always use absolute paths internally. I think the latter would be more foolproof, though the former is probably more ergonomic.

[mlissner]

The fix, I suppose, would be to either always use relative or always use absolute paths internally. I think the latter would be more foolproof, though the former is probably more ergonomic.

Sorry, I take this back, it's not about ergonomics. Users should be able to provide either relative or absolute paths and Pelican should handle them properly. Internally it can do whatever it wants, if it works. :)

even if you put a relative one into your command.

And I'll also note, in case it's not obvious, that using an absolute path in your --write-selected command also doesn't work (for the same reason), so there's really no workaround for this that I can come up with.

[mlissner]

Sorry, one other thought while I'm on this topic. If we're in this code fixing things anyway, would it be crazy to add support for globs to the write selected code? I find it really tiresome to have to track down paths, and it'd be much easier if I could just do:

--write-selected 'output/2019/*'

That'd reduce my ten year old blog's generation time by 90% and is much easier than tracking down this:

--write-selected 'output/2019/12/22/some-really-long-slug/index.html'

You can even do:

--write-selected 'output/2019/12/22/*'

And be pretty sure that you've found the one correct page to generate, if you fill the above with today's date.

Ok, that's all for me today, sorry for the many messages.

[avaris]

I didn't actually test this but looks like a bug. From your description, probably some kind of mismatch in paths has occurred along the way. Code dealing with paths in pelican is, let's say, "not ideal". Now that Python2 support is dropped this might be a nice opportunity to clean and unify stuff dealing with path and move to pathlib.

Glob support would be nice.

But a warning, using this as a way to reduce build time can be tricky. For one, --write-selected does what it states (at least when it was working). It generates output for the specified files, and only those files. i.e. giving --write-selected 'output/2019/*' would not update anything outside that folder with any new content, such as index, archives etc.

Any volunteers for a PR? :)

[mlissner]

Any volunteers for a PR? :)

I wish I could. I considered it before filing the bug, but I'm already up to my ears both professionally and on the side. I'm also pretty bad at filesystem stuff, which I guess is a bad excuse, because I should get better at it, but I'm sticking with it for today. :)

[flothesof]

Hi, thanks for opening this issue.
I've been wanting to use --write-selected too and have run into what I think is the same behavior: instead of just generating the specified file, it generates the whole blog every time. I'm now relieved it's not just me who doesn't get it to work...

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your participation and understanding.

[justinmayer]

@smartass101: As the original author of this feature, do you have any thoughts on how this issue might be resolved?

[hipe]

TL:DR; this patch fixes it for me

When I first encountered it in the Site Generation section, this was among the many attractive features of pelican that compelled me to want to migrate my hundreds of markdown files from my current format/SSG, so i was ... excited to get this feature working once i saw that it just silently fails

The work by @mlissner above got me on the right track

As suggested above, the "problem" is that paths get absolute-ized (in the place where the settings config dict is written to) but when the paths come in as arguments to this one function to decide whether to generate them, the are relative (eg output/foo/bar.html)

This patch fixes it for me, which is to simply (hackishly?) not absolute-ize the path as represented in config (command-arguments or in the config file). (The only thing the new code does now is set the config value to [] as a default.)

diff --git a/pelican/settings.py b/pelican/settings.py
index ea3ee8eb..52e4aae2 100644
--- a/pelican/settings.py
+++ b/pelican/settings.py
@@ -524,11 +524,9 @@ def configure_settings(settings):
             raise Exception("Could not find the theme %s"
                             % settings['THEME'])

-    # make paths selected for writing absolute if necessary
-    settings['WRITE_SELECTED'] = [
-        os.path.abspath(path) for path in
-        settings.get('WRITE_SELECTED', DEFAULT_CONFIG['WRITE_SELECTED'])
-    ]
+    # (used to absolute-ize)
+    if 'WRITE_SELECTED' not in settings:
+        settings['WRITE_SELECTED'] = DEFAULT_CONFIG['WRITE_SELECTED']

     # standardize strings to lowercase strings
     for key in ['DEFAULT_LANG']:

Having read over the associated issues linked to above by @avaris, i see that it is an area of some debate how which kinds of of paths are to be represented and transformed. This patch does not address any of those suggestions

I am not interested in the feature request for glob arguments to be supported, as that is well out of this scope of fixing this bug. (Use fnmatch probably) I may be interested in contributing to that as a feature, but again, out of scope for this bug

This passed 235 tests (18 skipped b.c i have no pandoc)

I have not yet determine why the existing unit test for this feature didn't catch that it's silently failing.. not sure if I will make time for this just now...

Cheers!

[hipe]

small update on this issue - it looks like it is even more complicated than I first thought:

if you pass an --output option, that path gets absolutized, but if you DON'T, the output path defaults to output (relative path); SO even with my patch, this is again broken if you pass an --output option.

Whether the internal representation of the output path is absolute or relative determines whether or not this feature is broken (and which way it is broken depends on whether you applied my above patch lol), so the bottom line is, at the least this case of passing or not passing an --output option needs to be added to the unit tests for this feature

More later..

[hipe]

lol yeah the TL;DR: of this for me for now is, if you're trying to use the --write-selected option, the workaround is you've also gotta use the --output option (and forget about my patch)

[leemengtaiwan]

I tried using --output with --write-selected option, but can't make single doc update work..

Also, I don't know how to apply the patch hack @hipe kindly provided above. when I search for pelican, it only showed me the pelican bin 😢 and suggestion or advice? 🙏

which pelican
my_conda_env/bin/pelican

[leemengtaiwan]

NVM That's a stupid question. I haven't update pelican version for years.
I will try to clone the pelican repo and build from there and see how things go.

[hipe]

@leemengtaiwan
maybe it's that you need to use an absolute path in --output

mkdir temp
cd temp
virtualenv -p python3 my_conda_env
source my_conda_env/bin/activate
pip install 'pelican[markdown]'
pelican-quickstart
[i said 'n' to every Y/n]

echo "Title: my file one\nDate: 2010-12-03 10:20\n## my header 1\n- item one\n- item two" > content/article-1.md
echo "Title: my file two\nDate: 2010-12-03 10:20\n## my header 1\n- item one\n- item two" > content/article-2.md

pelican --output=output --write-selected="/Users/hipe/path/to/pwd/output/my-file-two.html"

ls output/*.html      # output/my-file-two.html

let me know!

[leemengtaiwan]

@hipe still no luck with absolute path in --output but your clear example inspired me to write a script that does a little trick to achieve what I need:

1. copy current working md (e.g., content/article-1.md) into another dummy content path working_content
2. override pelican content path with that dummy content path, i.e., pelican working_content

And I can make my normal 30 seconds build time back to 1 second!!
It's hacky but efficient so maybe I will just go with this trick for now 😅

[mirekdlugosz]

But a warning, using this as a way to reduce build time can be tricky.

It seems that even at it's best, WRITE_SELECTED just doesn't provide significant time reduction to begin with - because rendering template and writing file to disk is not where Pelican spends most of the time. At this point, all files are already read, parsed, file system metadata was read (to fill gaps from front matter) and most content-modifying plugins have run.

Case in point: I changed is_selected_for_writing to do partial matches and set WRITE_SELECTED to match single article. I reduced build time from 0.91 second to... 0.85 second (as reported by pelican). That's reduction by some 6%.

I also played with idea of skipping reading of content based on source path. Build time was reduced from 0.91 second to 0.13 second. That's reduction by 85%.

Personally, I think that instead of fixing WRITE_SELECTED, it's better to remove it and introduce new way of selecting content based on source path. I might play with that idea more and contribute a patch, but I can't commit to any time frame. So if anyone else wants to grab this idea and run with it, feel free to do so.