User.js: Just cloning repo leads to two modified files prefsCleaner.bat and updater.bat

[sciamano]

Hi,
does anybody else has problems with line endings on linux?
I'm running linux and do not have any setting w.r.t. line endings in git configuration.
When i do just a fresh clone with

export LANG=C
git clone https://github.com/ghacksuserjs/ghacks-user.js.git
cd ghacks-user.js

and then check the status, the *.bat files are marked as modified

git status
modified:   prefsCleaner.bat
modified:   updater.bat

even they seem to be dos files (e.g. when i open them in vim)

PS: I have other git repos with gitattributes and dos files on the same linux box and they all are fine...

[Thorin-Oakenpants]

I'm not very knowledgeable on these things but you can see our file - https://github.com/ghacksuserjs/ghacks-user.js/blob/master/.gitattributes

## * text=auto

*.js text=auto
*.md text=auto
*.yml text=auto
*.txt text=auto
*.sh text=auto
*.bat eol=crlf

*.png binary

.gitattributes export-ignore
*.yml export-ignore
wikipiki export-ignore

I have no idea what .gitattributes export-ignore means

Did a quick google for "cloning git changes EOL" (sans quotes)

• https://dzone.com/articles/git-showing-file-modified-even (oldish from 2014)

From here (from Oct 2017)

One simple solution is:

• make sure core.autocrlf is set to false for all repos:
  
  
  
  • git config --global core.autocrlf false
  
  
• re-clone your repo, and check no EOL conversion is done.

^^ but I have no idea if that means core.autocrlf is at your end, or we add it here.

[ArchangeGabriel]

I can confirm I have the same issue, and it made it very hard for me to do workflow based on rebasing my changes above master regularly. I’ll try that listed solution and report.

[Thorin-Oakenpants]

try again - earthlng did something - https://github.com/ghacksuserjs/ghacks-user.js/commit/43119694fbf9e678bdf8f0dd9ee346a89832c5a9

[sciamano]

thanks for quick reply, unfortunately the last change does not fix it for me...
there is also a github page about line endings https://help.github.com/articles/dealing-with-line-endings/
but those settings do not help as well.
What I belive is that running the commans (maybe from linux?)

$ echo "* text=auto" >.gitattributes  <---- that we already have
$ git add --renormalize .
$ git status        # Show files that will be normalized
$ git commit -m "Introduce end-of-line normalization"

from gitattributes man page should fix it and will try it once i get a windows box to test that we do not simply shift the problem to the other end, e.g. windows users :)

[claustromaniac]

I think the problem with 4311969 + @sciamano's suggestion is that GitHub doesn't normalize line endings to CRLF on checkout when we download the raw content directly, or at least it didn't use to (remember #314?). I'm not sure though, things might have changed since then.

Another possibility that might work:

* text=auto

*.js text
*.md text
*.yml text
*.txt text
*.sh text
*.bat -text

*.png binary

.gitattributes export-ignore
*.yml export-ignore
/wikipiki export-ignore

The minor drawback is that batchfiles would never be normalized, I think. Source.

@sciamano, since you say you have other Git repos with gitattributes and dos files and they are fine, can you see what is different in those gitattributes files?

BTW, I read older versions of Git had all sorts of issues with .gitattributes. Probably not relevant in this particular case, but worth keeping in mind.

[earthlng]

AFAIK we need eol=crlf for batch files so that the raw links and the batch files in the zip have CRLF line endings.
IDK why those 2 files show as modified and I have no idea if there's anything we can do about it. You'll probably have to work around it on your end.

[claustromaniac]

So, are you saying that when you commented out the * text=auto at the top in 464ffcd, that wasn't really necessary at the time? If that's so, I guess you're right then and .gitattributes has always worked as it should here. :man_facepalming:

Another couple of things that might help, maybe combined. You can always revert them if they don't help:

git config core.fileMode false
git config --global core.trustctime false

https://git-scm.com/docs/git-config

BTW the text attribute in *.bat text eol=CRLF is redundant according to https://www.git-scm.com/docs/gitattributes EDIT: but GitHub still uses them at the same time in the examples in the docs, it seems.

[sciamano]

seems i found the porblem with the files:
prefsCleaner.bat contains problematic lines
https://github.com/ghacksuserjs/ghacks-user.js/blob/master/prefsCleaner.bat#L68
https://github.com/ghacksuserjs/ghacks-user.js/blob/master/prefsCleaner.bat#L76
and updater.bat contains the line
https://github.com/ghacksuserjs/ghacks-user.js/blob/master/updater.bat#L207
The lines end with only \r instead of proper \r\n that you see as the red ^M symbol
So the commands

$ git add --renormalize .
$ git commit -m "Introduce end-of-line normalization"

should fix the broken lines

[sciamano]

you do not see ^M at the lines in check-outed files, because git fix will it during the checkout (i think), but then they start to differ from repository version and shown as modified

[claustromaniac]

I always wondered why GitHub was showing the CR characters in those lines. Now I just want to know how the hell that happened in the first place... It was probably my fault, but I don't have a clue as to how I messed that up at the time.

[earthlng]

WTF!! It's impossible to fix it with the online editor!! It just moves the problem to where there wasn't 1 before!

@sciamano can you please run these commands which I assume should then create a new Pull Request that I can merge? But please make sure you DO NOT run echo "* text=auto" >.gitattributes first

[sciamano]

@earthlng i'v prepared PR, please check that the *.bat files still work as expected

[earthlng]

Thanks. Now it's a mix of what it was before:

• the files seem to be stored with LF in the repo and they get converted to CRLF when I git clone them on Windows as they should (I think)
• the 2 files have CRLF within the zip download file :+1:
• BUT the raw links are now back to LF which is the problematic part. :-1:

I'll try @claustromaniac's suggestion with *.bat -text and just upload the file with CRLF, hopefully that will fix all the problems fingerscrossed

[claustromaniac]

No! I already tried that! xD It didn't work.

Lemme submit a PR.

EDIT: I was too late.

EDIT² : seems to have worked on your end, but now we'll have to be careful with line endings when we edit those files. I had managed to fix it and keep the *.bat text eol=CRLF attributes on my end, but meh.

[earthlng]

no it didn't work. ATM the raw links have LF instead of CRLF.
When I try to upload the file with CRLF endings it shows the ^M again in the preview. I'm lost!

What we need (I think!) is to upload the file with proper CRLF and somehow tell github to never EVER touch the line endings for batch files. Any idea how to do that?

[claustromaniac]

Let me know when you're done testing stuff xD I already solved it but it takes a while to submit the PR because it takes many steps.

[claustromaniac]

(and I have to start over every time you push commits)

[earthlng]

I re-uploaded the files with CRLF and the raw links and the zip file are okay again.

@sciamano can you git clone in a completely fresh folder again and check if you still get the problem with git status showing the files as modified ?

I think hope thanks to *.bat -test in .gitattributes the files should remain untouched and no longer show as modified. That's at least what the docs says - if that doesn't work IDK what else to do! ...

Unsetting the text attribute on a path tells Git not to attempt any end-of-line conversion upon checkin or checkout.

[earthlng]

@claustromaniac oh damn, sorry mate. But yeah I'm done making any changes myself for now. Have to go kill some folks in Quake to calm down :)

[Thorin-Oakenpants]

Have to go kill some folks in Quake to calm down :)

I used to be a mercenary, until I took an arrow to the knee

[claustromaniac]

Seeing as how the infamous ^M is still there, I'm gonna give it a go.. No commits to master allowed for a while!!! :laughing:

[claustromaniac]

What I did before doesn't work anymore. W.T.F!

Unreal. Will try a few things more later. Commit all you want in the meantime.

Well, at least I am now 110% certain whatever happened in the first place was definitely NOT my fault xD

[sciamano]

@earthlng @claustromaniac : that all is really confusing, i did fresh clone and it looks fine, and at the same time on github i see some ^M at the line ends of the files. so there is something else going on, maybe github frontend (syntax highlighting for example) is just broken...
Anyway, my original issue is gone :)

[earthlng]

Ok great, thanks for testing again.

I downloaded the 2 files again in all 3 possible ways (raw, zip, git) on Windows and looked at the lines with hexview that github marks as faulty but there's 100% nothing wrong with those lines.

It looks like github stumbles on delims^=^'^" with its MSDOS parser. That's the only thing all 4 lines have in common. Github thinks there's an unclosed " but there isn't - it's perfectly valid DOS code.

But anyway, thanks for bringing this up and I'm glad we were able to fix it.
I learned something new at least :)

[claustromaniac]

It looks like github stumbles on delims^=^'^" with its MSDOS parser.

I just put that theory to a test, and it seems to be spot-on!

It means it can be worked around natively (which would allow us to use *.bat text eol=CRLF in .gitattributes if we wanted). The workaround, in this case, is as simple as adding REM " at the end of each of the problematic lines, as you can see in this test branch in my fork. The trolling ^M markers are finally gone :relieved:

I can submit a PR with those changes if you want, although I guess there's no need for it anymore.

[claustromaniac]

Actually, I think updating .gitattributes in my fork automatically changed the line endings in the raw files without me doing anything else, so nah. Better to keep the *.bat -text. The workaround in the batchfiles would be merely a cosmetic improvement at this point.

[earthlng]

Thanks @claustromaniac but I don't really care about cosmetics. Let's just keep it the way it is now