Hugo: Temporary files not ignored when (re)building site

[obar]

What version of Hugo are you using (hugo version)?

v0.62.2-83E50184/extended linux/amd64

Does this issue reproduce with the latest release?

Yes

I wasn't sure why I'd often be seeing the live preview of my site break on saves, when the state on the filesystem was fine. Digging into the no such file or directory message, I see this has to do with Emacs lock files being followed where I don't believe they should be. The full message is in the form of:

Error: add site dependencies: load resources: loading templates: walk: Readdir: decorate: lstat $SITE_ROOT/layouts/partials/$USER@$MACHINE.$PID:$BOOT: no such file or directory

Of course the lock file is a symlink pointing to this path, with the prefix .# added to the file name in an unsaved, modified buffer.

This is true for both content and templating files, as long as a buffer is open with a modified version. This happens for both hugo and hugo server commands. It's easiest to reproduce by triggering a rebuild (running hugo or saving a change to a different file with hugo server running), and having a file open in emacs with a modified state so the lock symlink exists. It seems to also happen as a race condition when editing only a single file with hugo server running, where the rebuild reaches the lock symlink before emacs removes it.

I believe hugo server ignores events around temporary files so a rebuild isn't triggered, but does not ignore these files in the building of a site. I further believe that, well… that's the wrong behavior.

I have a fix that solves my issue, see imminent pull request.

[anoduck]

Brilliant work! Awesome Sauce!

[obar]

This bug has now fallen onto the third page of issues, but I believe it's an easy fix so would rather bump than let it get stale. It does affect people, reported both here and on discourse.

Curious about work-arounds that people are using. @kaushalmodi, @masasam, looks like you use emacs and hugo together—could you comment on your approaches to this?

It seems like vim users would be affected as well, which has been partially addressed for config files already but not for other files.

[kaushalmodi]

@obar May be because I choose to not create auto-backup files in the same dir as the original file? I customize the backup-directory-alist in my Emacs config.. all backup files are dumped in the /tmp/${USER}/.backups/.

could you comment on your approaches to this?

Also my content source is outside the content/ dir that Hugo watches for content. I edit/create my content in a separate content-org dir (which sits parallel to the content/ dir, and I use my ox-hugo package to export from Org to Markdown (which get created in the directory Hugo expects).

[masasam]

@obar I'm setting not to create a lock file as follows.

 (setq create-lockfiles nil)

[obar]

Thanks for the responses!

@kaushalmodi that's an interesting package, I should look into it. I believe the backups aren't such a concern. They do exist as files if hugo tries to process them, the error is caused by lockfiles which are implemented as symlinks to targets that don't exist. In your case that issue is sidestepped through not editing the files in place but rather generating them from a different path.

Should the docs warn emacs/vim users to configure their editors differently (a note with @masasam's customization of create-lockfiles for instance), or is it the wrong behavior for hugo to not be ignoring these files?

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. The resources of the Hugo team are limited, and so we are asking for your help.
If this is a bug and you can still reproduce this error on the master branch, please reply with all of the information you have about it in order to keep the issue open.
If this is a feature request, and you feel that it is still relevant and valuable, please tell us why.
This issue will automatically be closed in the near future if no further activity occurs. Thank you for all your contributions.

[obar]

This bug still reproduces per instructions in the first post. It has come up on discourse again within the past month. The work around for emacs users is to turn off lockfiles (see previous discussion in this thread), but that has other side effects for emacs users. I don't know about turning off temporary files in other editors.

Fortunately the patch doesn't need any updates, it works today on master (see the latest comment on the pull request).

Can I call in a fellow Canadian for help, could you review the pull request to close this bug @anthonyfok ? It's small and doesn't add any new logic so I would think the review should be painless… famous last words ;)

[bep]

So, I'm not adding a potentially breaking hard-coded filter to the filesystems.

What would solve this and be generally useful would be if I could configure:

[files]
ignore = ["assets/**", "content/**.tmp"]

Where "assets" etc. would match the "virtual component folder" in Hugo.

Or something, but that would obviously be more work.

[obar]

It seems like pre-1.0 is a decent time to introduce this, because the default ignore list should include temporary files generated by common editors and if this filter list is breaking then so would be another feature (although a configurable filter list could be changed more easily by the user).

I'm not wed to the solution I submitted; any fix to this would be great, but I can't see the case where it would break an existing site. Can you give an example? Most of these filters look at the extension and we can agree nobody is intentionally naming their to-be-processed files as my_post.md.swp :) The only filter on base of the file name itself (other than what we've discussed already in the pull request, the vim file 4913) involves excluding # and .# as a prefix, which wouldn't work for hugo content anyway as # is the separator for a url fragment.

With all that said, I'm looking forward to a fix for this regardless of what it looks like. Thanks again!

[garrison]

It would be nice if Hugo would ignore temporary files entirely. However, the biggest problem for me is that even once these temporary files disappear, the only way I have found to get hugo server to notice and successfully serve the site again is to restart it. This breaks my thought process and work flow, every time.

[obar]

@garrison Yes, the server crashes as it tries to process the temp file (in the emacs case, following a symlink to nowhere) so a restart is a must.

a potentially breaking hard-coded filter to the filesystems

@bep I might have missed something but I don't believe such a filter can break any sites (see my previous comment for reasoning). If that's sound, could this approach work as a stopgap until a more elaborate ignoring system is built?