User.js: override.js file no longer overriding user.js file in FF58+

[overdodactyl]

edit: 1413413 is the cause - Thorin

edit II: I submitted a bug report report (1416279) to hopefully address this - overdodactyl

edit III: bug was labeled WONTFIX

The code that implements this functionality has been removed. After sending an intent to remove email to dev.platform and discussing, we decided that it wasn't worth the effort of continuing to support as we re-implement the preference service.

edit IV: wiki link

---

Greetings,

In the past, I've been using an override.js file to override some preferences from user.js as suggested in Section 1.5 of the wiki.

Sometime within the past week (maybe two?), the settings in my override.js file are no longer being used to override the user.js file in Firefox Nightly.

I can reproduce this issue in a new Nightly profile, however, if I copy my user.js file and my preferences folder containing override.js to a new profile and use the stable version of Firefox, I do not experience this issue.

Has something changed? Can anyone else confirm they are experiencing this in Nightly (or, alternatively, that they aren't having the problem)?

I apologize if this has already been addressed...I did a quick search here and on bugzilla and couldn't find anything relevant .

[Thorin-Oakenpants]

That sucks. I haven't got 58 here to test with to confirm (not that I doubt you!). Wouldn't surprise me if some tweak to shave a ms eg lazy loading of prefs caused this. Or someone removed it in their line of work as it's undocumented.

Might be worth looking at all the bugzillas resolved for 58

[overdodactyl]

Thanks for the reply @Thorin-Oakenpants, I'll take a closer look at bugzilla and report back with anything I find!

Hopefully it wasn't removed on purpose...I've been relying on that :)

[Thorin-Oakenpants]

Personally, I only have a couple of changes, so I just add mine to the end of the user.js. But I was going to use this to introduce a relaxed version eg override-relaxed.js rather than fluffing around with all the extra work in another branch. I guess it's always possible to add a relaxed section to the existing user.js at the end, but a separate file would be better - so here's hoping this is a regression that can be fixed

Psst! To get all 58 tickets... go to release notes - eg v56, scroll to the bottom and near the footer is a "Complete list of changes for this release" - modify the linked URL and replace all instances of 56 with 58 (there should be three of them)

https://bugzilla.mozilla.org/buglist.cgi?j_top=OR&f1=target_milestone&o3=equals&v3=Firefox%2058&o1=equals&resolution=FIXED&o2=anyexact&query_format=advanced&f3=target_milestone&f2=cf_status_firefox58&bug_status=RESOLVED&bug_status=VERIFIED&bug_status=CLOSED&v1=mozilla58&v2=fixed%2Cverified&limit=0

[overdodactyl]

Thank you @Thorin-Oakenpants!! I appreciate the advice and you looking into that...I'm not super familiar with bugzilla, so any new tips like that are really helpful :) I'm glad a tickets already been submitted (one that I obviously missed).

I don' have too many overrides either, so throwing them at the end of user.js is a good solution in the mean time.

Thanks again!

[Thorin-Oakenpants]

I'm not hopeful. That ticket is for Thunderbird and is mainly about thunderbird's inbuilt Lightning extension, and the initial patch is hardcoded exceptions for that extension. Guess we'll see what happens.

PS: I updated the wiki page at the top

Update: OK, that ticket is toast. They are ONLY concerned about Thunderbird. Questions about Firefox have been hidden as off-topic. So unless someone opens a new ticket as a regression or whatnot from the original Bug 1413413, then I guess it's all over

[overdodactyl]

Thanks for updating the wiki page!

I noticed that was for Thunderbird...I guess I just naively thought the ticket might include Firefox haha. You're right though, that doesn't look super promising.

I can go ahead and submit a new ticket if you think it would be helpful.

[Thorin-Oakenpants]

^ That would be cool. Something along the lines of extensions stay exempt but allow users (as in people) to put their own files there - it only needs to be allowed during startup and only for read properties

[Thorin-Oakenpants]

@earthlng
This explains the mystery re the order preferences/*.js files were read. Since it was a directory specifically for extension-preferences, then the order wouldn't matter (assuming each extension stuck to their specific prefs and didn't sneak in FF ones)

[overdodactyl]

I went ahead and submitted a bug report this morning 1416279 for anyone that wants to follow it.

Edit: Was labeled as WONTFIX...I guess there's no hope after all :(

The code that implements this functionality has been removed. After sending an intent to remove email to dev.platform and discussing, we decided that it wasn't worth the effort of continuing to support as we re-implement the preference service.

[Thorin-Oakenpants]

@overdodactyl Thanks for trying. I guess we were way too late to the party / discussions .. before they ripped out all the code

[overdodactyl]

No problem @Thorin-Oakenpants, sorry I couldn't do more! If it helps anyone else out there, a viable alternative (at least for me) is to use something like the following script:

#!/bin/bash
curl -o .../profile/user.js https://raw.githubusercontent.com/ghacksuserjs/ghacks-user.js/master/user.js
cat .../profile/preferences/override.js >> .../profile/user.js 

it at least quickens the process of keeping up-to-date and could be easily modified to use various *.js files.

[Thorin-Oakenpants]

Yeah. I modified the wiki header to say that we recommend keeping your "override.js" as a master copy and just append to the user.js when needed via copypasta. For ease of use I said rename it to user-override and stuff it in the profile root dir

[earthlng]

Hi @overdodactyl

sorry I couldn't do more!

you found a problem, informed us about it, reported it to mozilla and provided a script that can be used as a workaround - what more could we ask for! Thanks for taking the time to do all that, much appreciated! ❤️❤️❤️

[Thorin-Oakenpants]

Yeah @overdodactyl ... :kiss: :kiss:

[claustromaniac]

I made myself this batch script (link removed) for Windows. I leave it here in case anyone wants it.

It backs up user.js and then it downloads the user.js file from this repository and appends _user-overrides.js_ to it.

It goes in the profile folder alongside user.js and user-overrides.js.

Thank you for this great resource, by the way. I've been using user.js files for a long time but this one is by far the most comprehensive I ever had.

Also thank you @overdodactyl for the heads up.

[Thorin-Oakenpants]

@claustromaniac thanks buddy, err pal ... cool avatar too :)

When I get time, I think I update the wiki with a new new page on auto-updating - I will post the actual batch as code (and also save a copy as a file in the repo). Would also be good to add instructions/equivalents for Linux and Mac (I don't know these)

Looking at the batch file, could we

• add an intro of text with a cancel option eg
  
  
  
  • [Title] ghacks user.js updater
    
    This batch should be run from your Firefox profile directory. If you have a user.js, it will be updated with the latest ghacks version from github and then append any of your own changes from user-override.js - Continue Y / N
  
  
• what happens if user.js.bak already exists
• can the script lookup the version number - its always line 4 and insert that somewhere before user confirms?

Edit:
sweeet... raw/master/user.js uses LF for line endings

[claustromaniac]

@Thorin-Oakenpants I improved it a bit with your ideas, and I changed a few things.

If user.js.bak already existed it just deleted it and created a new backup, but I changed that. It now creates a secondary backup, and when the update finishes it checks whether user.js and the new backup are the same or not and, if they are, it keeps the old (different) backup. Otherwise it keeps the new backup (and discards the older one).

Other significant change is that now it doesn't rely on the existence of user-overrides.js to update user.js.

I guess there's still room for improvements, so let me know what you think.

Edit: Last-minute minor fix 😓

[Thorin-Oakenpants]

Ooooh .. I'm getting a little wet .. looks great. I see it also pulls the local version number - nice!

I'll test it later, not sure when. I will be incommunicado in a few days for almost a week (no internet, no cell phone coverage), and no, it's not a secret special blacks ops mission - it's because the earth is huge and their are places with almost no humans and infrastructure (woohoo!). Am back on the 25th 26th, so sometime after that we can do a wiki page etc - unless earthlng gets industrious :) :kiss:

[overdodactyl]

Hi @earthlng

Thanks for taking the time to do all that, much appreciated!

It's the least I could do given how much I've benefited from this project :)

@Thorin-Oakenpants

I decided to try and make a version of @claustromaniac's script usable for fellow Mac users out there . I tried to incorporate all your suggestions from his first version, but I made this before I saw the most recent one. Feel free to use it, make any suggestions/change it however you like or completely ignore it all together. I'm pretty new to writing scripts, so this was a fun use of my time either way :)

#!/bin/bash

cd "`dirname $0`"

echo
echo "This script should be run from your Firefox profile directory."
echo

if [ -e user.js ]; then 
    echo "Your current user.js file for this profile will be backed up and the latest ghacks version from github will take its place."
    echo
    echo "If currently using the ghacks user.js, please compare versions:"

    online_version="$(curl -s https://raw.githubusercontent.com/ghacksuserjs/ghacks-user.js/master/user.js | sed -n '4p')"
    local_version="$(sed -n '4p' user.js)"

    echo "  Available online: ${online_version}"
    echo "  Currently using:  ${local_version}"

else    
    echo "A user.js file does not exist in the current directory.  If you continue, the latest ghacks version from github will be downloaded"
fi
echo
echo "If a user-override.js file exists in this directory, it will be appended to user.js."
echo
read -p "Continue Y/N? " -n 1 -r
echo
echo

if [[ $REPLY =~ ^[Yy]$ ]]; then
    if [ -e user.js ]; then
        #backup current user.js
        mv user.js "user.js.backup.$(date +"%Y-%m-%d_%H%M")"
        echo "Your previous user.js file was backed up: user.js.backup.$(date +"%Y-%m-%d")"
    fi

    #download latest ghacks user.js
    echo "downloading latest ghacks user.js file"
    curl -O https://raw.githubusercontent.com/ghacksuserjs/ghacks-user.js/master/user.js
    echo "ghacks user.js has been downloaded"

    if [ -e user-overrides.js ]; then
        echo "user-overrides.js file found"
        cat user-overrides.js >> user.js
        echo "user-overrides.js has been appended to user.js"
    fi
else
    echo "Process aborted"
fi

[Thorin-Oakenpants]

I'm not an expert on these scripts. Kinda thinking now how cool a little cross platform exe would be :) That aside, this is all great. I am sure you can all nut it out and test things before or for when I get back - and then we can host files and write a wiki page :+1:

[earthlng]

Awesome stuff guys!! Thanks @claustromaniac + @overdodactyl ! ❤️

nits:
In the mac (+linux?) version you're downloading the file twice (not that it matters - just sayin' :) and in certain places it would be better to use && to make sure the previous command succeeded. I would also store the backup filename in a variable so that you can show the actual filename:

#backup current user.js
bakfile="user.js.backup.$(date +"%Y-%m-%d_%H%M")"
mv user.js "${bakfile}" && echo "Your previous user.js file was backed up: ${bakfile}"

[earthlng]

I tested the Windows batch script and noticed that I got this 1 weird line () in the concatenated user.js due to the BOM header because my user-overrides.js was stored as UTF-8. IDK if that's a problem when FF will parse the file and/or whether cat on Mac/Linux will also create that line when the 2 files are not in the same encoding. Just something to keep in mind I guess and maybe something that we should add a note about somewhere because some people might use UTF-8 because of foreign language comments in their user-overrides.js. The raw user.js on github is 1252 ANSI latin-1 encoded.
Users either need to use ANSI encoding or UTF-8 without the BOM for their user-overrides.js file.

[earthlng]

https://github.com/ghacksuserjs/ghacks-user.js/blob/master/updater.bat
https://github.com/ghacksuserjs/ghacks-user.js/blob/master/updater.sh

This will help users a lot, thanks guys!!

[overdodactyl]

@earthlng, I'm glad I could contribute a bit (even if it was something small)!

Thanks for cleaning up my code a bit, that was my first script of any real value over a couple lines, so I figured it was far from perfect :)

I noticed in some of my profiles, the script wasn't locating to the correct directory before running.

I changed cd "dirname $0" to cd "$( dirname "${BASH_SOURCE[0]}" )" && pwd and it seemed to work, but I can't completely explain why...

On a totally unrelated note, I noticed there was a comma missing on line 25 after 'media.webspeech.recognition.enable' in the script for bulk resetting preferences. (thought it was worth pointing out but not worthy of it's own issue :) )

edit: my bad, fixed the comma - thorin

[claustromaniac]

@Thorin-Oakenpants I must admit that I also thought about compiling an exe, but that would bring along the hassle of dependencies and such, right? I mean, for a cross-platform approach we would need something like Python. It would be nice if someone could test @overdodactyl's script on Linux, though. Since it's not overly complex it might just work fine on many distros.

@earthlng Good catch. I wouldn't hope for Firefox to be expecting some people using user.js files made out of chunks of text with different encoding, so I guess we're indeed better off just avoiding that 😆 It's good to know that it seems it's just the BOM header that is leaving trash in the resulting file.

[earthlng]

@overdodactyl
fix for the profile detection problem: https://github.com/ghacksuserjs/ghacks-user.js/commit/500c129d484494d06ef3cffcc91f534077e5dfc3
It's based on info I found online but I don't have a Mac and can't test it. Let me know if it doesn't work.

[claustromaniac]

So... I made some further changes and additions to the batch script. 😸

• It can now handle read-only files.
• it is somewhat more explicit regarding what it's doing in some circumstances. For example, it now informs the user when no changes are made.
• It now accepts two parameters: -unattended and -log
• Minor improvements here and there.

As you might have guessed, the -unattended parameter skips the parts where user input would normally be required for the script to continue. On the other hand, the -log parameter redirects the script's output to user.js-update-log.txt incrementally. The parameters can be added to a shortcut's path field, or when setting up the script to run as a scheduled task.

I'm quite satisfied with it as it is now, but please make suggestions/requests/observations if you have any.

EDIT: Fixed the new logging feature messing with the script output when disabled 🤦‍♂️

[earthlng]

Very nice changes. I've reviewed + tested it and can confirm it works as advertised :) :+1:
As for suggestions/requests/observations maybe add >nul to the powershell command? What do you think? Other than that I can't think of anything else to make this any better, good stuff! Thanks

[claustromaniac]

I finally taught myself to open pull requests and just opened one to change some minor details that I noticed today lol

I enjoy small projects like this once in a while, so it's been my pleasure, really. I'm the one who is grateful for all that you guys do for us here.

[earthlng]

I enjoy small projects like this once in a while

My words exactly and now I'm stuck here xD Anyway, thanks a lot for contributing

[claustromaniac]

Some other potential improvements I've come up with that may be worth considering:

• Remove comments from the resulting user.js file. This would reduce file size to about a 10% of the original size, and could be added as an optional feature with a switch like -stripcomments.
• Open a changelog (or the like) after a successful update. This could also be optional and could happen only if changes are made. The script would take the user to a static URL to read the release notes, or maybe the URL could be inside the user.js file somewhere. Or something like that. There could be other ways to do it.
• Allow merging multiple override files with a common prefix/suffix.
• Make the -unattended switch run the script completely silent (hiding the console, or minimising it). Unfortunately this would require a separate VBS or third-party software. Or building the entire thing from scratch in another language. Or maybe not.. I'll test something later.
• Have the updater auto update itself? xD Probably too much.

[earthlng]

Allow merging multiple override files with a common prefix/suffix.

I like this one. Like fe. (re-)create the user-overrides.js from all the .js files in the preferences folder. That would be neat. The other ideas sound complicated to implement, fe re: changelog - you already end up with a new user.js and the backup and can simply compare the 2 to see all the changes.

[claustromaniac]

(re-)create the user-overrides.js from all the .js files in the preferences folder

Actually, I was thinking more of something like appending all user-something.js files to user.js alphabetically, without modifying any other files, because user.js is the only one Firefox cares about. It would mean that you can have as many separate override files as you want and they will all count.

Maybe a mix of both concepts could be achieved if the script merged all js files in a specific subfolder into a new user-overrides.js file and then appended that one to user.js, but then user-overrides.js would be redundant.

changelog - you already end up with a new user.js and the backup and can simply compare the 2 to see all the changes.

The point of having the script bring up a changelog upon completion would be simply automation, and it would go hand in hand with the option to run it unattended. But you are right. It's probably not worth it.

As for some of my other ideas:

Make the -unattended switch run the script completely silent

Apparently, the task scheduler can hide the console window when both the _Run whether user is logged on or not_ and _Hidden_ options are selected. A great reason to not even bother.

Have the updater auto update itself?

The logic goes like this: Introducing an updater means taking away some of the hassle of updating files, but it isn't all that much beneficial if you now have to manually update the updater. Ideally, one would simply write a functional updater and leave it alone, but you never really know if the future will demand making further changes to it or not.

What I can't make up my mind about is the how(s). I mean, there are many ways to _code_ it, but _when_ should the updater attempt to update itself? Should it be optional? How?

EDIT: I can't help but feel a little bit like this while I think of all this. Just a little bit.

[Thorin-Oakenpants]

Pssst! @claustromaniac ... go here .. https://github.com/ghacksuserjs/ghacks-user.js/invitations .. you have an invite to join all the cool kids .. this means anyone can just type @ and easily get your name. Plus it makes you look awesome (speaking from experience). I think it also automagically adds the repo to your watch list, but you can just toggle that off after accepting the invite, if the incessant emails are too much :)

[Thorin-Oakenpants]

:doh: @overdodactyl And you too buddy ^^ see above link

[overdodactyl]

@earthlng:

fix for the profile detection problem: 500c129
It's based on info I found online but I don't have a Mac and can't test it. Let me know if it doesn't work.

I apologize for the delayed response, I've been away from the computer for the past several days. The only issue with using greadlink on a Mac is that it's not a native command. In order for it to work, you have to first install the _coreutils_ package:

brew install coreutils

Does the approach mentioned in my previous comment (cd "$( dirname "${BASH_SOURCE[0]}" )" && pwd) work on a linux system (from reading online, I _think_ it should)? If so, I think this might be a better approach as Mac users wouldn't have to install any extra dependencies.

@Thorin-Oakenpants

Thanks for the invite!!

I need to catch myself up with @claustromaniac - both with improvements to the script and learning to submit pull requests :)

Edit: Here is a link discussing readlink/greadlink on a mac:

http://biercoff.com/fixing-readlink-illegal-option-f-error-on-a-mac/

[earthlng]

@claustromaniac

EDIT: I can't help but feel a little bit like this while I think of all this. Just a little bit.

LOL, so true! I feel like that a lot xD

Maybe a mix of both concepts could be achieved if the script merged all js files in a specific subfolder into a new user-overrides.js file and then appended that one to user.js, but then user-overrides.js would be redundant.

If someone has multiple js overrides I just think it would be nicer to have them all in a subfolder.

@overdodactyl

greadlink on a Mac is that it's not a native command

oh okay, that's not great. I think it needs some sort of readlink command to work in all cases. I don't think your suggested code does that. From what I remember there can be problems if you call it with ./updater.sh and also when using a symlink. They mentioned stat as an alternative I think. Will try to find the link where I read that.

[earthlng]

@overdodactyl can you test if the change in my PR works for you, please? PR: https://github.com/ghacksuserjs/ghacks-user.js/pull/284

[overdodactyl]

@earthlng I didn't have luck with PR 285, unfortunately. I tried a different approach based off here and had success in all the profiles I tested. I tried submitting my first pull request with the changes: PR 285.

[earthlng]

Your PR adds a function realpath_osx() but calls realpath. (?)
I changed my PR to fallback to what you suggested (cd "$( dirname "${BASH_SOURCE[0]}" )") and merged it. It doesn't work in all cases but it should be good enough for most people.

[overdodactyl]

I was just looking into what you mentioned earlier with stat, I'll let you know if I figure something out. The previous use of just "${BASH_SOURCE[0]}" as the alternative to readlink and greadlink results in user.js being put in the home directory instead of the profile directory.

It doesn't work in all cases

Sounds like it might get a little whacky with the use of symlinks?

Your PR adds a function realpath_osx() but calls realpath. (?)

My mistake. When I was testing it I used realpath, then when I copied it over to github I thought it would be good to change it to realpath_osx to make it more clear what it was used for...forgot to change it in both places.

[claustromaniac]

I need to catch myself up with @claustromaniac - both with improvements to the script and learning to submit pull requests :)

@overdodactyl For your own good, do not try to catch up. It's too late for me, but you can still run away. Do it while you still can, before you begin the quick descent into madness.

[claustromaniac]

I haven't tested the latest version of my script exhaustively, so I might have missed something.

Be sure to let me know if you guys find anything odd, or if you'd like me to explain in more detail how it works. I know the code itself is awful to read, but that was part of the charm. I don't have too much experience with batch scripts, so it was a fun challenge.

[earthlng]

I gave up trying to fix the missing !s and ^s, I just can't get it to work. For user_pref lines in override files there are easy workarounds to keep the lines un-merged, fe by prepending a /* don't merge this */ or simply a TAB. We can mention that in the wiki. Pants and I just have to remember not to use those characters in comments behind user_pref lines.

[claustromaniac]

@earthlng I see a lot has been going on while I wasn't looking heh.

I still haven't given up on that. To be honest, I didn't have time to test what I wanted to test yet.

As for the loopìng auto-update part, it's pretty easy to fix. I'll submit another PR soon (hopefully).

[Thorin-Oakenpants]

Pants and I just have to remember not to use those characters in comments behind user_pref lines.

If you are appending to the user.js, then surely everything in it up to the last line is ignored, as in read only. That is, you grab the ghacks user.js, download it, go to last EOL and start adding. I'm confused, but then again I have not been paying attention to exactly what you two have been doing.

Where do we have these ! (count:30) and ^ (count: zero) chars, can we remove them in the user.js (bit hacky) and what about the ones people put in their override.js (we can't control the contents of that). Do they have to be on a line starting with "user_pref" ?

Edit: 26 of the 30 uses of ! in the user.js are the parrot pref lines

[earthlng]

If you have a user-overrides.js and run the script without any parameters it will simply append your overrides to the end of the user.js. If you instead have multiple override .js files in profile/preferences you can rename that folder to user.js-overrides and run the script with the -multioverrides switch and it will append all those files to the end of the user.js.

And then there's the -merge switch which will do an in-place replacement of active user_pref lines. for example the original user.js has a line

user_pref("whatever", true);

and the user-overrides.js has

user_pref("whatever", false); /* 1234 */

with -merge, the final user.js will have that line replaced with the exact line from the overrides.
-merge can be used with a single user-overrides.js or in combination with -multioverrides for multiple files.

Do they have to be on a line starting with "user_pref" ?

yes. But if someone wants to use the -merge switch and also likes to have comments with !s or ^s, they can either add a TAB or a JS-multi-line comment in front of the user_pref line, fe.

/* don't merge this */ user_pref("whatever", false); // !!! THIS NEEDS TO BE FALSE, WTF !!!

This will keep that line at the bottom of the user.js, together with any other overrides that didn't get merged, for example because a certain pref does not exist or is inactive in the user.js.

user.js.parrot lines are NOT merged and can contain those 2 characters.

All we have to do, is not use ! or ^ in comments behind user_pref lines.

[Thorin-Oakenpants]

All we have to do, is not use ! or ^ in comments behind user_pref lines.

Well "we" (ghacks user.js) comply with that - can't speak for people's override *.js files

PS: I assigned you the wiki writing task :+1:

[claustromaniac]

@earthlng Actually, after some testing and reading I found out carets ( ^ ) are unaffected.

The issue with exclamation marks is that they are removed if found anywhere in user_pref lines (except _user.js.parrot lines because those are never merged), and if there are at least two exclamation marks in the same line, everything in between them will get removed as well.

I still haven't had any luck fixing it.

[earthlng]

Yeah I don't think we need to bother with trying to fix it, we'll just note it in the wiki.

[Thorin-Oakenpants]

going to close this, use #283 (you guys seem to be doing most of your chats in the PRs anyway). When the wiki page is done we can update the user.js as well. Meanwhile, the PSA on this is only 3 weeks old (is it enough?) but the current old wiki page method covers it anyway