Lint-staged: Lost new untracked files

Created on 26 Jan 2020 · 27Comments · Source: okonet/lint-staged

Hello there.

For the second time I experience very strange issue which leads to total lose of unstaged files :((((

I had some changes and few new files. I did:

git add
git commit

After running all the hooks I can see that all my new files are lost.

Thanks in advance for any help

Here is log from console:

cast-call-client on  integrate-tailwindcss [$✘!?] via ⬢ v12.14.1
➜ gst                                                                                                       21:12:54
On branch integrate-tailwindcss
Changes not staged for commit:
  (use "git add/rm <file>..." to update what will be committed)
  (use "git checkout -- <file>..." to discard changes in working directory)

    modified:   .eslintrc
    modified:   package-lock.json
    modified:   packages/client/.flowconfig
    deleted:    packages/client/.storybook/addons.js
    deleted:    packages/client/.storybook/config.js
    deleted:    packages/client/flow-typed/npm/styled-components_v3.x.x.js
    modified:   packages/client/package-lock.json
    modified:   packages/client/package.json
    modified:   packages/client/src/app.jsx
    deleted:    packages/client/src/assets/castcalls-logo.png
    deleted:    packages/client/src/assets/logo_black.png
    modified:   packages/client/src/components/Header.jsx
    modified:   packages/client/src/components/portfolio/PortfolioGrid.stories.jsx
    modified:   packages/client/src/components/portfolio/PortfolioPreview.stories.jsx
    modified:   packages/client/src/components/portfolio/PortfolioTile.stories.jsx
    modified:   packages/client/src/components/portfolio/PortfolioTileWithPreview.stories.jsx
    modified:   packages/client/src/components/portfolio/portfolio-tile/PortfolioTileInfo.stories.jsx
    modified:   packages/client/src/components/subscriptions/stories/InvolvementCard.stories.jsx
    modified:   packages/client/src/components/subscriptions/subscription-grid/SubscriptionGrid.stories.jsx
    modified:   packages/client/src/components/subscriptions/subscription-grid/components/SubscriptionPortfolioTile.stories.jsx
    modified:   packages/client/src/index.jsx
    modified:   packages/client/src/pages/landing/LandingPage.jsx
    modified:   packages/client/src/router/MainRouter.jsx
    modified:   packages/client/src/style-config.js

Untracked files:
  (use "git add <file>..." to include in what will be committed)

    packages/client/.storybook/main.js
    packages/client/.storybook/preview.js
    packages/client/flow-typed/npm/classnames_v2.x.x.js
    packages/client/postcss.config.js
    packages/client/src/assets/Logos/
    packages/client/src/cc-kit/
    packages/client/src/components/Header.stories.jsx
    packages/client/src/tailwind-out.css
    packages/client/src/tailwind.config.js
    packages/client/src/tailwind.css
    packages/client/src/utils/MediaMatcher.js
    packages/tests/package-lock.json

no changes added to commit (use "git add" and/or "git commit -a")

cast-call-client on  integrate-tailwindcss [$✘!?] via ⬢ v12.14.1
➜ ga packages/client/package.json                                                                           21:12:55

cast-call-client on  integrate-tailwindcss [$✘!+?] via ⬢ v12.14.1
➜ gst                                                                                                       21:13:18
On branch integrate-tailwindcss
Changes to be committed:
  (use "git reset HEAD <file>..." to unstage)

    modified:   packages/client/package.json

Changes not staged for commit:
  (use "git add/rm <file>..." to update what will be committed)
  (use "git checkout -- <file>..." to discard changes in working directory)

    modified:   .eslintrc
    modified:   package-lock.json
    modified:   packages/client/.flowconfig
    deleted:    packages/client/.storybook/addons.js
    deleted:    packages/client/.storybook/config.js
    deleted:    packages/client/flow-typed/npm/styled-components_v3.x.x.js
    modified:   packages/client/package-lock.json
    modified:   packages/client/src/app.jsx
    deleted:    packages/client/src/assets/castcalls-logo.png
    deleted:    packages/client/src/assets/logo_black.png
    modified:   packages/client/src/components/Header.jsx
    modified:   packages/client/src/components/portfolio/PortfolioGrid.stories.jsx
    modified:   packages/client/src/components/portfolio/PortfolioPreview.stories.jsx
    modified:   packages/client/src/components/portfolio/PortfolioTile.stories.jsx
    modified:   packages/client/src/components/portfolio/PortfolioTileWithPreview.stories.jsx
    modified:   packages/client/src/components/portfolio/portfolio-tile/PortfolioTileInfo.stories.jsx
    modified:   packages/client/src/components/subscriptions/stories/InvolvementCard.stories.jsx
    modified:   packages/client/src/components/subscriptions/subscription-grid/SubscriptionGrid.stories.jsx
    modified:   packages/client/src/components/subscriptions/subscription-grid/components/SubscriptionPortfolioTile.stories.jsx
    modified:   packages/client/src/index.jsx
    modified:   packages/client/src/pages/landing/LandingPage.jsx
    modified:   packages/client/src/router/MainRouter.jsx
    modified:   packages/client/src/style-config.js

Untracked files:
  (use "git add <file>..." to include in what will be committed)

    packages/client/.storybook/main.js
    packages/client/.storybook/preview.js
    packages/client/flow-typed/npm/classnames_v2.x.x.js
    packages/client/postcss.config.js
    packages/client/src/assets/Logos/
    packages/client/src/cc-kit/
    packages/client/src/components/Header.stories.jsx
    packages/client/src/tailwind-out.css
    packages/client/src/tailwind.config.js
    packages/client/src/tailwind.css
    packages/client/src/utils/MediaMatcher.js
    packages/tests/package-lock.json


cast-call-client on  integrate-tailwindcss [$✘!+?] via ⬢ v12.14.1
➜ gc -m 'integrate-tailwindcss'                                                                             21:13:19
husky > pre-commit (node v12.14.1)
  ✔ Preparing...
  ✔ Running tasks...
  ✔ Applying modifications...
  ✔ Cleaning up...
[integrate-tailwindcss 2c1a81f] integrate-tailwindcss
 1 file changed, 15 insertions(+), 4 deletions(-)

cast-call-client on  integrate-tailwindcss [$✘!] via ⬢ v12.14.1 took 2s
➜ gst                                                                                                       21:13:25
On branch integrate-tailwindcss
Changes not staged for commit:
  (use "git add/rm <file>..." to update what will be committed)
  (use "git checkout -- <file>..." to discard changes in working directory)

    modified:   .eslintrc
    modified:   package-lock.json
integrate-tailwindcss
    modified:   packages/client/.flowconfig
    deleted:    packages/client/.storybook/addons.js
    deleted:    packages/client/.storybook/config.js
    deleted:    packages/client/flow-typed/npm/styled-components_v3.x.x.js
    modified:   packages/client/package-lock.json
    modified:   packages/client/src/app.jsx
    deleted:    packages/client/src/assets/castcalls-logo.png
    deleted:    packages/client/src/assets/logo_black.png
    modified:   packages/client/src/components/Header.jsx
    modified:   packages/client/src/components/portfolio/PortfolioGrid.stories.jsx
    modified:   packages/client/src/components/portfolio/PortfolioPreview.stories.jsx
    modified:   packages/client/src/components/portfolio/PortfolioTile.stories.jsx
    modified:   packages/client/src/components/portfolio/PortfolioTileWithPreview.stories.jsx
    modified:   packages/client/src/components/portfolio/portfolio-tile/PortfolioTileInfo.stories.jsx
    modified:   packages/client/src/components/subscriptions/stories/InvolvementCard.stories.jsx
    modified:   packages/client/src/components/subscriptions/subscription-grid/SubscriptionGrid.stories.jsx
    modified:   packages/client/src/components/subscriptions/subscription-grid/components/SubscriptionPortfolioTile.stories.jsx
    modified:   packages/client/src/index.jsx
    modified:   packages/client/src/pages/landing/LandingPage.jsx
    modified:   packages/client/src/router/MainRouter.jsx
    modified:   packages/client/src/style-config.js

no changes added to commit (use "git add" and/or "git commit -a")

Environment

OS: macOs Catalina 10.15.1
Node.js: 12.14.1
lint-staged: 10.0.0

released

Source

MikeYermolayev

👍4

Most helpful comment

:tada: This issue has been resolved in version 10.0.3 :tada:

The release is available on:

Your semantic-release bot :package::rocket:

okonet on 27 Jan 2020

❤5

All 27 comments

This definitely shouldn't happen.

What's your git version? I see this is a monorepo, are you running lint-staged in a subdirectory or the main git repo root directory? Can you run using the --debug flag and post the output?

The untracked files are hidden into the main backup stash, using git stash --include-untracked --keep-index and restored after the tasks have run. If they're not back, there might be something wrong in this part.

Can you add an untracked file, and then run these commands:

git  stash save --include-untracked --keep-index
git stash pop

If the untracked file is back, then it might not be a direct Git issue but something else.

iiroj on 26 Jan 2020

Thanks for the fast reply

git version 2.21.0 (Apple Git-122)

Yes, it's a monorepo and lint-staged is set up on the root

MikeYermolayev on 26 Jan 2020

Maybe you know some way to restore those files? A little bit afraid to do something with 'stash' is there is a chance to restore them. Lost work results from few days

MikeYermolayev on 26 Jan 2020

The command git reflog might help with that.

iiroj on 26 Jan 2020

I'm also experiencing this. I've been trying to figure out why this is happening, and as far as I could understand, I think this is happening because when executing the stashBackup, the unstagedDiff will just store the already existing files (git diff [--options] stash@{n} will not display nothing for untracked files since they are on the third parent of a stash commit). In order to add the untracked files code to unstagedDiff, we can do it by executing git rev-list -g stash | git rev-list --stdin --max-parents=0 | xargs git diff --binary --unified=0 --no-color --no-ext-diff --patch.

(This is undesired behaviour is not happening in version v10.0.0-beta.1).

kratos-42 on 27 Jan 2020

After upgrading to version 10, I'm experiencing this as well. I lost my entirely untracked files. Unfortunately, VSCode keeps the files open so that I can save it again. But the static files I made are all gone.

In order to get the untracked files code we can do it by executing git rev-list -g stash | git rev-list --stdin --max-parents=0 | xargs git diff --binary --unified=0 --no-color --no-ext-diff --patch.

Hi @kratos-42 , when I run that command, it returns

I'm reverting back to use version 9.

jefrydco on 27 Jan 2020

@jefrydco when I said to run that command I was proposing a solution to solve the bug, and not providing a way to get your files back. I'm sorry if I was not clear.
(Btw, you are getting that error probably because your stash is empty).

kratos-42 on 27 Jan 2020

👍1

I’m terribly sorry about this bug, and I’ve no idea where it comes from since it hasn’t been an issue during development.

I’ll try to get this fixed ASAP. The first thing to do is throw errors when restoring untracked files fails, so the process fails and everything gets restored.

Are there a lot of untracked files when this happens? The tests only use a single untracked file so maybe it’s maybe it’s related to the size of the diff (speculating based on the first report’s _git status_).

iiroj on 27 Jan 2020

I don't think there is a failure in restoring to be thrown. I think the problem is in the response from the git diff because it does not include the untracked files code, so when executing git apply the only changes it will apply/restore are just for the files that already exists (remotely or in a previous commit).

It happens no matter if there's just one or many untracked files.

kratos-42 on 27 Jan 2020

The untracked files are restored separately from the this.unstagedDiff here:

https://github.com/okonet/lint-staged/blob/293547d46080eac007393709a0d63a63f2063fff/lib/gitWorkflow.js#L200

This happens after running tasks and applying back the unstaged diff. The untracked files are taken straight from the third commit.

iiroj on 27 Jan 2020

@kratos-42 Oke, got it. My stash files aren't empty anyway.

Are there a lot of untracked files when this happens? The tests only use a single untracked file so maybe it’s maybe it’s related to the size of the diff (speculating based on the first report’s git status).

I think the @kratos-42 opinion was right. It will apply/restore are just for the files that already exist. My unstaged files are mostly just created and there was a lot of them.

jefrydco on 27 Jan 2020

The untracked diff might not have taken binary files into acocunt, because the --binary flag was missing. Could this be the reason?

iiroj on 27 Jan 2020

I don't know what kind of binary files you mean, but my untracked files were consists of markdown, images and video files. The files that already exist were unfortunately opened in VS Code so I can save it again. But the images and video were all gone.

jefrydco on 27 Jan 2020

The untracked diff might not have taken binary files into acocunt, because the --binary flag was missing. Could this be the reason?

I tried using the --binary flag and it indeed worked for .js files. However, for an image it did not work...

kratos-42 on 27 Jan 2020

I'm afraid the error may have been caused by the combination of the missing --binary flag and the empty catch {}. These in combination would not cause _lint-staged_ to abort when the patch failed to apply.

I've opened https://github.com/okonet/lint-staged/pull/780 for this issue.

iiroj on 27 Jan 2020

👍2

Sorry man, but I don't think that is enough to solve the problem. Using --binary the git show output is fine, but the changes are not being applied as expected. The binary files are still being deleted and no error is being thrown.

kratos-42 on 27 Jan 2020

@kratos-42 If you're able to debug further, are the binary files in the diff, or missing?

I'll try this:

add new untracked binary file
create a stash with --include-untracked
get the diff
apply it
???
see if binary file is back

EDIT: This works at least when manually running the commands. The major difference is that I'm piping the output of the diff to git apply, vs. _lint-staged_ saves it in a variable.

iiroj on 27 Jan 2020

@kratos-42 If you're able to debug further, are the binary files in the diff, or missing?

The binary files are in the diff.

kratos-42 on 27 Jan 2020

@kratos-42 In the PR, if the git apply step fails, it now throws an error. This will cause _lint-staged_ to do a git reset and return to the original state by applying the entire backup stash. In this sense it would fix the current behaviour.

However, it would be nice to figure out why it doesn't apply.

iiroj on 27 Jan 2020

The command git reflog might help with that.

Tried it but haven't found any info about lost files in commits here (

MikeYermolayev on 27 Jan 2020

However, it would be nice to figure out why it doesn't apply.

Well, a possibly an error with execa input option?

The major difference is that I'm piping the output of the diff to git apply, vs. lint-staged saves it in a variable.

A way to do this could be redirecting the output to a file and read from there, something like:

await execa(
  'git',
  [
    'show',
    ...GIT_DIFF_ARGS,
    '--format=%b',
    `${backupStash}^3`
  ],
  { all: true, cwd: process.cwd() }
).stdout.pipe(fs.createWriteStream('stdout.patch'));

await this.execGit([...GIT_APPLY_ARGS, 'stdout.patch']);

kratos-42 on 27 Jan 2020

@MikeYermolayev Maybe read this if it contains something that can help:
https://stackoverflow.com/questions/89332/how-to-recover-a-dropped-stash-in-git

@kratos-42 Thanks for the suggestion. Since we create two diffs (unstaged, untracked), I'd then probably redirect both of those to files. The files should probably be deleted right after applying.

iiroj on 27 Jan 2020

👍5

Thanks a lot!

Your link helped and I was able to find my files pretty easily by using
gitk --all $( git fsck --no-reflog | awk '/dangling commit/ {print $3}' )

MikeYermolayev on 27 Jan 2020

🎉5

@kratos-42 based on your suggestion, I pushed new commits to https://github.com/okonet/lint-staged/pull/780 that, instead of using variables inside the Node.js process, saves the unstaged and untracked diffs temporarily inside the .git directory (so as to not mess up git status). The files are then deleted in the cleanup step.

Can you perhaps test this yourself?

iiroj on 27 Jan 2020

🎉2

@iiroj Everything is perfectly working 👌

kratos-42 on 27 Jan 2020

@kratos-42 Very good! We'll assume that one contributing factor was indeed the very large size of the diff.

iiroj on 27 Jan 2020

🎉1

:tada: This issue has been resolved in version 10.0.3 :tada:

The release is available on:

Your semantic-release bot :package::rocket:

okonet on 27 Jan 2020

❤5

Was this page helpful?

0 / 5 - 0 ratings

Related issues

Updating to lint-staged v9 issue -- using .eslintignore throws warnings!

tnrich · 7Comments

Roadmap

okonet · 5Comments

On v10.0.0-0, committing changes to a JS file from Github Desktop fails

acusti · 6Comments

lint-staged is not taking .eslintrc.js configuration.

pumanitro · 7Comments

Allowing non-zero exit code in some of the commands to support pre-commit autofixing

kachkaev · 5Comments