Doom-emacs: Revert project text search to previous behavior

[Cerebus]

What did you expect to happen?
Project text search should search normal files.

Project text search should search hidden files with the universal argument.

What actually happened?
Project text search searches hidden files by default.

Additional details:
Revert commit a904b718f888bb94207035c6599688a05c72507e

rg searches text in hidden files only with an explicit flag. Doom should do the same to keep user expectations in line. Searching hidden files when invoked with the universal argument aligns with user expectations.

Steps to reproduce:
In a project with hidden files:

1. SPC /
2. Note --hidden on the command line.

System information:

SYSTEM  type       darwin
        config     x86_64-apple-darwin19.6.0
        shell      /bin/zsh
        uname      Darwin 19.6.0 Darwin Kernel Version 19.6.0: Mon Aug 31 22:12:52 PDT 2020; root:xnu-6153.141.2~1/RELEASE_X86_64 x86_64
        path       (~/.local/share/virtualenvs/pynet-analysis/bin /usr/local/bin /usr/bin /bin /usr/sbin /sbin /Library/TeX/texbin /usr/local/munki /opt/X11/bin ~/Library/Python/Current/bin /usr/local/sbin ~/.local/bin ~/bin /usr/local/go/bin ~/projects/go/bin ~/projects/personal/go/bin)
EMACS   dir        ~/.emacs.d/
        version    28.0.50
        build      Oct 28, 2020
        buildopts  --disable-dependency-tracking --disable-silent-rules --enable-locallisppath=/usr/local/share/emacs/site-lisp --infodir=/usr/local/Cellar/emacs-plus@28/28.0.50/share/info/emacs --prefix=/usr/local/Cellar/emacs-plus@28/28.0.50 --with-xml2 --with-gnutls --without-dbus --with-imagemagick --with-modules --with-rsvg --with-ns --disable-ns-self-contained
        features   JPEG TIFF GIF PNG RSVG IMAGEMAGICK GLIB NOTIFY KQUEUE ACL GNUTLS LIBXML2 ZLIB TOOLKIT_SCROLL_BARS NS MODULES THREADS JSON PDUMPER LCMS2
        traits     (server-running envvar-file)
DOOM    dir        ~/.doom.d/
        version    2.0.9
        build      HEAD -> develop 57ef63d6 2020-10-29 05:42:10 -0400
        elc-files  0
        modules    (:completion company ivy :ui doom doom-dashboard doom-quit hl-todo indent-guides minimap modeline ophints (popup +defaults) treemacs vc-gutter vi-tilde-fringe workspaces :editor (evil +everywhere) file-templates fold (format +onsave) snippets word-wrap :emacs dired electric undo vc :term eshell :checkers syntax :tools ansible docker (eval +overlay) lookup (lsp +eglot) (magit +forge) pass terraform :lang emacs-lisp (go +lsp) (haskell +dante) json markdown (org +journal) plantuml (python +lsp) sh yaml :email mu4e :config (default +bindings +smartparens))
        packages   ((base16-theme) (auto-virtualenv) (pyvenv) (kubernetes) (groovy-mode))
        unpin      (n/a)
        elpa       (n/a)

[magnetophon]

I think it's good to include hidden files by default, but maybe we could have some mechanism to exclude .git and other common dirs one wants to exclude.

[robbert-vdh]

I agree that --hidden plus a blacklist for common VCS directories (.git, .svn, .hg etc.) would be a good default. Project search in my dotfiles is mostly useless now because everything is inside of .config or .local directories, and using -uu or plain --hidden will also search all git objects which is probably never what you'd actually want to do.

[Cerebus]

I can't put is more succinctly than this:

• Text search tools (rg, the GNU grep family, and so on) skip hidden files unless they are explicitly included.
• User expectation with any search tool is that it will behave similarly to other tools used in the past.
• Searching hidden files by default is not how similar tools behave, so user expectation is violated.
• Violating user expectations is a UX design flaw.

Putting --hidden behind C-u would be The Emacs Way.

If you want to change the _project file search_ to use --hidden by default, there's a better argument there b/c tools like find look directly at inodes and don't care about shell-based semantics like hidden files. However, this discussion is specific to text searching.

[robbert-vdh]

While that is the default behaviour of grep and ripgrep, when doing a project search you'd expect that to search the entire project, right? No other text editors or IDEs I've used with a 'Find in project...' option ignore dotfiles.

[ineu]

when doing a project search you'd expect that to search the entire project, right?

No, I wouldn't expect it to search in my git history at least. It is sometimes convenient to search in dotfiles (e.g. .env files), but these cases are rare compared to the normal search. Dotfiles are hidden for reason, one doesn't need them too often.

Now I tend to fall back to using rg in the shell because the new behavior adds too much garbage to the results (mostly from .git). I believe the proposed 'C-u' for '--hidden' is the right approach, but even if '--hidden' is going to stay as the default, it should at least be configurable via an option. At the moment it's hardcoded in +ivy-file-search. I'm not an elisp expert, but I believe with the current approach the only way to get rid of '--hidden' is to override the whole function.

[The-Compiler]

My $0.02: On various occasions, I was confused because my CI configurations in .github/workflows/ (for GitHub Actions) weren't being searched - so +1 on using --hidden but ignoring VCS directories (like @robbert-vdh said, that's what I'd expect from a "project search" in a text editor or IDE).

[otavioschwanck]

I agree that --hidden plus a blacklist for common VCS directories (.git, .svn, .hg node_modules etc) would be a good default. If is possible, ignore the files on .gitignore too.

[The-Compiler]

ripgrep already reads .gitignore (even with --hidden), so I think things like node_modules would be covered (also, note that --hidden won't have changed anything about node_modules). Obviously .gitignore is git-specific, it also reads .ignore which seems to be a standard between different search tools (also supported by e.g. ag).

Thus, I'd say the only thing which would need to be ignored by default with --hidden are directories which are hidden, contain noise, but are not in a .gitignore by default - from what I can see, that means only .git. If you use another VCS, I think it's okay to assume you'd have an .ignore file to use search tools, and that file would then already contain .svn or so - but someone correct me if I'm wrong, I've used git wherever possible for the past 8 years or so.

Thus, I'd propose a commandline like ag ... --hidden --glob='!.git' - does that sound sensible enough, with my rationale about not including other VCS directories above? If so, I can try submitting a PR. If the commandline gets longer, I guess it'd make sense to share the commandline arguments between the two places somehow, but I don't speak enough Lisp to do that.

[axelson]

A temporary workaround for this that has been working for me is to add a /.git line to each project's .gitignore (from what I can see there aren't any side-effects of this, but I didn't do any actual research).

[hlissner]

Sorry for the delay. As others have mentioned, there are dotfiles that we may want to search; particularly configuration files, but searching .git is unanimously undesirable, so I've omitted it explicitly. Meanwhile, fd/rg should still omit ignored files (via .gitignore or .ignore).

As of 51744ce I think we've reached a good compromise and this issue should be resolved. Let me know if that isn't the case and I'll reopen it. Thanks for bringing it to my attention (and sorry for any trouble it may have caused)!