Todo-tree: Extended globs

[muuvmuuv]

It looks like I can not use more than ** as regex in globs, but I would like to archive something like this:

Note the last lines, where I want to allow these paths but ignore the rest

{
  "todo-tree.globs": [
    "!**/.git/**",
    "!**/node_modules/**",
    "!**/vendor/**",
    "!**/tmp/**",
    "!**/cache/**",
    "!**/logs/**",
    "!administrator/**",
    "!bin/**",
    "!cli/**",
    "!components/**",
    "!images/**",
    "!includes/**",
    "!jdownloads/**",
    "!language/**",
    "!layouts/**",
    "!libraries/**",
    "!media/**",
    "!modules/**",
    "!plugins/**",
    "modules/mod_additive_*/**",
    "plugins/content/additive_*/**",
    "components/com_additive/**"
  ],
}

[Gruntfuggly]

Hi - the globs are passed to ripgrep, so it's possible there is a limitation there.

Could you enable the debug console (set todo-tree.debug to true) and restart. Then in the Output window, you should find a channel for todo-tree in the drop down.

In there you should see the command that is generated to search the files. You could either try copying the command and running it yourself in a terminal window, or paste it here so I can try something similar.

[muuvmuuv]

It looks like it does not understand if there is not a slash or **. I'll have a look into RG docs, Thanks :)

→ rg --no-messages -H --column --line-number --color never  -e "((//|#|<-exec plutil -convert xml1 && grep -l "Phrase" {}|;|/\*|^)\s*(TODO|REMOVED|ADDED|EDITED|FIXED|NOTE|ERROR|BUG|EDITSTART|EDITEND)|^\s*- \[ \])" -g "plutil -convert xml1 && grep -l "Phrase" {} \;*/.git/**" -g "plutil -convert xml1 && grep -l "Phrase" {} \;*/node_modules/**" -g "plutil -convert xml1 && grep -l "Phrase" {} \;*/vendor/**" -g "plutil -convert xml1 && grep -l "Phrase" {} \;*/tmp/**" -g "plutil -convert xml1 && grep -l "Phrase" {} \;*/cache/**" -g "plutil -convert xml1 && grep -l "Phrase" {} \;*/logs/**" -g "plutil -convert xml1 && grep -l "Phrase" {} \;*/libs/**" -g "plutil -convert xml1 && grep -l "Phrase" {} \;*/dist/**" -g "plutil -convert xml1 && grep -l "Phrase" {} \;*/assets/**" -g "*" -g "ts3server -s*" -g "*" -g "!components/**" -g "!images/**" -g "!includes/**" -g "!jdownloads/**" -g "!language/**" -g "!layouts/**" -g "!libraries/**" -g "!media/**" -g "!modules/**" -g "!plugins/**" -g "modules/mod_additive_*/**" -g "plugins/content/additive_*/**" -g "components/com_additive/**" "/Users/marvinheilemann/Documents/Customers/Additive/Websites/additive/development"
zsh: event not found: components/

[muuvmuuv]

This seems to work in my terminal: rg -w TODO -g 'modules/mod_additive_*/*' -g '!./modules/**' -g 'plugins/content/additive_*/*' -g 'components/com_add*/*' -g 'templates/**' -g '!**/node_modules/**' -g '!**/dist/**'

note the single quotes as suggest by ripgrep

I tried this in VSCode with no result:

{
"todo-tree.globs": [
    "modules/mod_additive_*/*",
    "!./modules/**",
    "plugins/content/additive_*/*",
    "components/com_add*/*",
    "templates/**",
    "!**/node_modules/**",
    "!**/dist/**"
  ]
}

[Gruntfuggly]

Would you be prepared to try out a change for me to see if it works?
I you go edit .vscode/extensions/Gruntfuggly.todo-tree-0.0.78/ripgrep.js
at line 102, change it from

return `${command} -g \"${glob}\"`;

to

return `${command} -g \'${glob}\'`;

then reload the window and see if it makes any difference?

Also, I notice that your generated command contains things like -g "plutil -convert xml1 && grep -l "Phrase" {} \;*/node_modules/**" which doesn't appear in your globs definition - has it made a complete mess of generating the globs?

The command should simply contain a -g <glob> for each element of your globs array, so if other stuff is appearing then I've done something wrong!

There may be a problem where it is doing the string substitution...

[Gruntfuggly]

Rather than doing the string substitution as it's done at the moment, you could also try the simpler

return command + " -g \'" + glob + "\'";

I'm not sure why it did it the first way. I doubt it will fix anything, but it might be worth a try.

[muuvmuuv]

Nope, that is not working @Gruntfuggly.

And yes, I'm sorry for this, my iTerm generated this plutil-things, just ignore them.

Here a complete log output:

btw: why it is searching the user settings? My open files: ripgrep.js, index.php, User Settings

Search: {"file":"/Users/marvinheilemann/Library/Application Support/Code/User/settings.json"}
Command: /Applications/Visual\ Studio\ Code.app/Contents/Resources/app/node_modules.asar.unpacked/vscode-ripgrep/bin/rg --no-messages -H --column --line-number --color never  -e "((//|#|<!--|;|/\*|^)\s*(TODO|REMOVED|ADDED|EDITED|FIXED|NOTE|ERROR|BUG|EDITSTART|EDITEND)|^\s*- \[ \])" -g '!./modules/**' -g 'modules/mod_additive_*/**' -g 'plugins/content/additive_*/**' -g 'components/com_add*/**' -g 'templates/additive/**' -g '!**/node_modules/**' -g '!**/dist/**' -g '!**/libs/**' "/Users/marvinheilemann/Library/Application Support/Code/User/settings.json"
Found 0 items
onDidChangeActiveTextEditor (uri:{"$mid":1,"external":"output:extension-output-%232","path":"extension-output-#2","scheme":"output"})
onDidChangeActiveTextEditor (uri:{"$mid":1,"fsPath":"/Users/marvinheilemann/Documents/Customers/Additive/Websites/additive/development/templates/additive/index.php","external":"file:///Users/marvinheilemann/Documents/Customers/Additive/Websites/additive/development/templates/additive/index.php","path":"/Users/marvinheilemann/Documents/Customers/Additive/Websites/additive/development/templates/additive/index.php","scheme":"file"})
Search: {"folder":"/Users/marvinheilemann/Documents/Customers/Additive/Websites/additive/development"}
Command: /Applications/Visual\ Studio\ Code.app/Contents/Resources/app/node_modules.asar.unpacked/vscode-ripgrep/bin/rg --no-messages -H --column --line-number --color never  -e "((//|#|<!--|;|/\*|^)\s*(TODO|REMOVED|ADDED|EDITED|FIXED|NOTE|ERROR|BUG|EDITSTART|EDITEND)|^\s*- \[ \])" -g '!./modules/**' -g 'modules/mod_additive_*/**' -g 'plugins/content/additive_*/**' -g 'components/com_add*/**' -g 'templates/additive/**' -g '!**/node_modules/**' -g '!**/dist/**' -g '!**/libs/**' "/Users/marvinheilemann/Documents/Customers/Additive/Websites/additive/development"
Search: {"file":"/Users/marvinheilemann/Library/Application Support/Code/User/settings.json"}
Command: /Applications/Visual\ Studio\ Code.app/Contents/Resources/app/node_modules.asar.unpacked/vscode-ripgrep/bin/rg --no-messages -H --column --line-number --color never  -e "((//|#|<!--|;|/\*|^)\s*(TODO|REMOVED|ADDED|EDITED|FIXED|NOTE|ERROR|BUG|EDITSTART|EDITEND)|^\s*- \[ \])" -g '!./modules/**' -g 'modules/mod_additive_*/**' -g 'plugins/content/additive_*/**' -g 'components/com_add*/**' -g 'templates/additive/**' -g '!**/node_modules/**' -g '!**/dist/**' -g '!**/libs/**' "/Users/marvinheilemann/Library/Application Support/Code/User/settings.json"
Found 0 items

[Gruntfuggly]

OK - I'll see if I can recreate the problem with some globs of my own.

It searches the workspace folder and also any open files which aren't in the workspace folder.

[muuvmuuv]

Okay, I tried something else now: copied command from debug console > escaped the ! in the regex > executed this:

rg --no-messages -H --column --line-number --color never  -e "((//|#|<\!--|;|/\*|^)\s*(TODO|REMOVED|ADDED|EDITED|FIXED|NOTE|ERROR|BUG|EDITSTART|EDITEND)|^\s*- \[ \])" -g '!./modules/**' -g '!./plugins/**' -g '!./components/**' -g '!./templates/**' -g 'modules/mod_additive_*/**' -g 'plugins/content/additive_*/**' -g 'components/com_add*/**' -g 'templates/additive/**' -g '!**/node_modules/**' -g '!**/dist/**' -g '!**/libs/**' -g '!**/cache/**' "/Users/marvinheilemann/Documents/Customers/Additive/Websites/additive/development"

That seems to work for my installation in the terminal. When I try to change the regex for your extension in the settings "todo-tree.regex": "((//|#|<\\!--|;|/\\*|^)\\s*($TAGS)|^\\s*- \\[ \\])", I get this error:

Error parsing regex near '|#|<\!--|;' at character offset 9: Unrecognized escape sequence: '\!'.
(Hint: Try the --fixed-strings flag to search for a literal string.)

And it makes no difference if I use my local or the VSCode rg:

Local 0.9.0
VSCode: 0.8.1

[Gruntfuggly]

I don't think you should need to escape the ! in the regex. As far as I know, ! is only used with lookbehinds and ripgrep doesn't support those anyway.

Did escaping it on the command line fix the problems with your globs?

[muuvmuuv]

Yes, when I escape the ! in the command line it works

[Gruntfuggly]

So, are you saying that when you escape the ! in the regex, that the globs work as expected?

[muuvmuuv]

@Gruntfuggly correct. But only in the command line. Strange...

_{Sent with GitHawk}

[Gruntfuggly]

Did you get any further with this?

One thing you could try is replacing the ! with \x21 instead of escaping.

I think the reason escaping makes it work on the command line is that the ! may have been doing some variable substitution or something in the shell.

[muuvmuuv]

@Gruntfuggly nope, honestly I must say I'm now using todo+ and highlight together. It is more complex but I got it working.

[Gruntfuggly]

OK, no worries. I'm going to try and extend the globs functionality because I want to support the file watching functionality and vscode's basic globbing doesn't work very well. I'll probably separate the include abd exclude globs to make it simpler.

I'll post on this thread when it's done - it'll be up to you if you want to try it. :grinning:

[muuvmuuv]

@Gruntfuggly sounds good! I'll definitely try it, cause' I love the easiness of your extension!

[muuvmuuv]

I finally got it working in another way. I added "todo-tree.rootFolder": "/Users/marvinheilemann/Documents/Additive/Websites/additive/development/templates/additive", and this seems to just show me results from my templates folder. Anyway I hope the globs will work someday to be more specific because I also want to have two other folders included.

PS: Is it the correct behaviour that todo-tree.rootFolder needs an absolute path? It would be more efficient if it just uses node path to be able to use relative and absolute, IMHO.

[Gruntfuggly]

To run the search, ripgrep needs a 'working' folder which is currently set to '/', which means that absolute paths are needed at the moment. However, you can also use environment variables and ${workspaceFolder}.

I could add a setting to allow the working folder to be set to something else?

My current plan is to change the current todo-tree.globs into a list of exclude globs and then have a separate setting for 'include' globs (or folders).

[Gruntfuggly]

BTW - I just had a thought. Have you tried adding '**' to the start of your glob patterns?

E.g. **/templates/**/*

[muuvmuuv]

My current plan is to change the current todo-tree.globs into a list of exclude globs and then have a separate setting for 'include' globs (or folders).

Great idea @Gruntfuggly!

Your idea with **/templates/**/* is kind of working (removed rootFolder): it shows only templates todos but also shows node_modules todos. Tried to exclude them with !**/node_modules/** with no result.

[Gruntfuggly]

Globs seem to be much harder to define than you'd think. 😞

[Gruntfuggly]

I've just uploaded a new version with separate include and exclude globs - these globs are applied after the search, so there's no need to pass the globs to ripgrep anymore.

Let me know if it works for you. In theory you should be able to use your original glob definitions - the exclude globs don't need the ! in front of them though.

[muuvmuuv]

@Gruntfuggly DUUDE! This finally works! Super smooth, big THANKS!

My final file:

{
  "todo-tree.excludeGlobs": [
    "**/node_modules/**", "**/vendor/**", "**/bower_components/**", "**/dist/**"
  ],
  "todo-tree.includeGlobs": [
    "**/administrator/components/com_additive/**/*", "**/components/com_additive/**/*", "**/modules/mod_additive_*/**/*", "**/plugins/content/additive_*/**/*", "**/templates/**/*"
  ],
}