This is Request Control | GitHub
This is our Request Control Filters Wiki
Post your Request Control Filter-fu here
Note: Examples below are for discussion / testing - use/test at your leisure, post feedback. As really cool filters emerge, we will put them in the wiki
crssi
All 42 comments
NOTE: ga_ stuff might need XHR type also
Pure URL extension replacement:
Pattern: Any URL
Types: Document, Embedded document (not sure for second one)
Action: Filter
Filter URL: Off
Trim URL Parameters: ref_, utm_source, utm_medium, utm_term, utm_content, utm_campaign, utm_reader, utm_place, ga_source, ga_medium, ga_term, ga_content, ga_campaign, ga_place, yclid, _openstat, fb_action_ids, fb_action_types, fb_ref, fb_source, action_object_map, action_type_map, action_ref_map, ws_ab_test, btsid, algo_expid, algo_pvid, sid, utm_name, utm_cid, utm_reader, utm_viz_id, utm_pubreferrer, utm_swu, icid, _hsenc, _hsmi, mkt_tok, sr_share, vero_conv, vero_id, nr_email_referer, ncid
UPDATE:
Added @Atavic suggested "Trim URL Parameters"
crssi
on 21 Jun 2017
Redirect to URL without REF tracking:
Pattern: Any URL
Types: Document, Embedded document (not sure for second one)
Action: Redirect
Redirect To: {href/(.?)ref=./$1} {href/(.*\/)ref=.*/$1}
Samples:
https://www.amazon.com/AmazonBasics-Type-C-USB-Male-Cable/dp/B01GGKYQ02/ref=sr_1_1?s=amazonbasics&srs=10112675011&ie=UTF8&qid=1489067885&sr=8-1&keywords=usb-c
crssi
on 21 Jun 2017
Remove the crap and possible tracking over URL manipulation after images and CSS:
Pattern scheme: http/https
Pattern host: *
Pattern path: *.jpg?*, *.gif?*, *.png?*, *.svg?*, *.css?*
Types: Document, Embedded document (not sure for second one), Stylesheet, Image
Action: Filter
Filter URL: Off
Trim URL Parameters: Trim all
UPDATE:
Added Pattern path: *.css?*
Stylesheet breaks this page https://technet.microsoft.com/en-us/library/2009.07.cableguy.aspx
Added Types: Document, Embedded document
Samples:
http://www.24ur.com/
Sample that doesn't work, but it should... don know why, yet: UPDATE: It works now.
https://thechive.files.wordpress.com/2017/06/dd0523324267789bb8beecc5d7914970.jpg?quality=85&strip=info&w=600
BRAINING: maybe there is same method to be added also for Font and Media types... will see.
crssi
on 21 Jun 2017
UPDATE: Use "Skip Redirect" from AMO... its way better.
Facebook redirect without visiting facebook:
Pattern scheme: http/https
Pattern host: l.facebook.com
Pattern path: *u=*
Types: Document, Embedded document (not sure for second one)
Action: Filter
Filter URL: On
crssi
on 21 Jun 2017
Funny... I have to look into it too.
Anyway, I have updated for CSS too... see change up.
One site is particulary full of this crap to test... see updated post.
crssi
on 21 Jun 2017
Hah... resolved, done.
Have phun
crssi
on 21 Jun 2017
UPDATE: import/export is done.
The author is working now on import/export. ;)
@Thorin-Oakenpants - if you wish to clean the thread, you can delete my last 3 posts (including this one) and your "didn't work" too, since its resolved.
Cheers
crssi
on 21 Jun 2017
Here some Google's Urchin Tracking Modules that aren't listed in Trim URL Parameters list:
utm_name
utm_cid
utm_reader
utm_viz_id
utm_pubreferrer
utm_swu
Also, url-tracking-stripper has a few more for other trackers:
ICID
icid
_hsenc
_hsmi
mkt_tok
sr_share
vero_conv
vero_id
nr_email_referer
ncid
Atavic
on 22 Jun 2017
Thanks @crssi !! :+1:
a couple nits if you don't mind...
- the ga_ stuff probably needs to include XHR
{href/(.?)ref=./$1}doesn't seem right. I'm pretty sure this always results in a singleh(from "http/https"), right?(.?)
and wouldn't it (if fixed) break something like http://foo.bar/redirect?href=bar.foo/index.html
to give you an example, this is what I came up with for a bing-redirect:
{search/^\?.*&?q=(.+?)(&.*|#.*|$)/https://www.ixquick.eu/do/dsearch?query=$1&cat=web&pl=opensearch&language=english} - it covers everything I could think of, namely:
- h ttp://www.bing.com/search ?q=crssi &go=Submit+Query&qs=ds&form=QBLH
- h ttp://www.bing.com/search?go=Submit+Query &q=crssi &qs=ds&form=QBLH
- h ttp://www.bing.com/search?go=Submit+Query&qs=ds&form=QBLH &q=crssi
- h ttp://www.bing.com/search?q=crssi
- h ttp://www.bing.com/search?go=Submit+Query&qs=ds&form=QBLH &q=crssi#anchor1
and even then, I'm sure I missed something - regexes are almost impossible to get right
another one from my small collection:
Any URLCSP reportBlock
earthlng
on 22 Jun 2017
You are right. Need to rethink the REF one, it was fishy to me also and I don't like it at all.... will put a "Under construction" note on that post.
The ga_ stuff, should we separate into another filter extended with XHR or just enable the XHR on the current one. I will enable here now and see if I will get any breakage... will also put a note in that post.
I have another "nasty" one, which could replace "Skip Redirect" extension, but it is also fishy... for now I am using Redirector, since I can use regex (not that I like it, but can't see better way.
I can place the nasty one in another post together with the Redirector approach for your critics?
crssi
on 22 Jun 2017
Have you installed and looked at Redirector yet?
nope
Does Request Control offer anything that Redirector doesn't or vice versa?
IDK, you tell me :) I'll try it someday. My way too complex bing regex is probably not necessary. It looks like the addon strips additional parameters automatically. I'll try to come up with a simpler solution
earthlng
on 22 Jun 2017
I already edited my post dude - chillax mate :)
I'll probably end up using both
earthlng
on 22 Jun 2017
Redirector has some problems, when clicking URL from external program it doesn't always work as intended.
Another thingy is that Redirector parses input only by one filter (just my observation) and doesn't pass the result over filters again unitl nothing left to filter (it happens that you want to filter out more than only one thing).
Otherwise it has also exclusions which is great and better string manipulation over regex.
Both are very good.
crssi
on 22 Jun 2017
@Thorin-Oakenpants do you have any sample for ga_ stuff, so I can dig in?
After tomorrow I will be offline for about 5 days.
crssi
on 22 Jun 2017
does not cover @Thorin-Oakenpants sample: http://foo.bar/redirect?href=bar.foo/index.html
crssi
on 22 Jun 2017
UPDATE: Use "Skip Redirect" from AMO... its way better.
Now a nasty one... trying to mimic part of "Skip redirect"
NOTE: this one can and will make some login breakage... for tests and brainstorming only
Does not interfere any of rules posted on this page.
It consists 2 filter rules.
First one does the skipping and the second one whitelists login pages to avoid breakage.
Pattern: Any URL
Types: Document
Action: Filter
Filter URL: On
Pattern scheme: http/https
Pattern host: *
Pattern path: *signin*, *signout*, *login*, *logout*, *logon*, *logoff*, *auth*, *account*, *eBayISAPI.dll*, *ServiceLogin*, *ServiceLogout*, *AccountChooser* (maybe you can add also *option*, but I don't like it)
Types: Document
Action: Whitelist
Samples:
Working redirect
https://outgoing.prod.mozaws.net/v1/b928e4237edbbdd2646a3971d2e6b514aee033c10f3f4c49415bf93096405f38/http%3A//www.google.com/chrome/%3Fi-would-rather-use-firefox=http%253A%252F%252Fwww.mozilla.org/
www.google.com/chrome/?i-would-rather-use-firefox=http%3A%2F%2Fwww.mozilla.org/
Currently broken login pages
https://login.leagueoflegends.com/?region=eune&lang=en_PL&redirect_uri=http%3A%2F%2Feune.leagueoflegends.com%2F
I can't figure out how to match this one since "login" is part of hostname and not path and Request Control doesn't cover this... here Redirector is better.
crssi
on 22 Jun 2017
Crap... at least "path" definition is case sensitive. I have placed an issue about that.
UPDATE: Case sensitive "bug" is actually "deal breaker" until corrected. Shame, started to love this one. :(
crssi
on 22 Jun 2017
:-1: Redirector always loads the unmodified address.
Maybe we should add Request Control to Appendix B: Firefox-Add-ons?
Atavic
on 22 Jun 2017
^^ Ahh, this now perfectly explains my observation about Redirector and what I didn't like about it.
But how then uBo, uMatrix and Request Controls are dealing perfectly in those cases.
crssi
on 22 Jun 2017
URL path definition is case sensitive
The absolute URL path: http://example.com/data/ is not case sensitive, as per RFC 3986.
While all that comes after can be case sensitive, depending on web server settings, OS...
Everything after the RFC Standard is unclear.
Atavic
on 22 Jun 2017
@crssi wrote:
do you have any sample for ga_ stuff
any site that uses GoogleAnalytics, fe ghacks. GA should already be blocked by uBO or uMatrix or whatnot and I'm not sure we really need a filter for that stuff.
earthlng wrote
My way too complex bing regex is probably not necessary. It looks like the addon strips additional parameters automatically
Yeah, it doesn't, at least not for "Action: Redirect". Redirector doesn't make it easier either.
But I realized I don't need to account for anchors because they are not part of the "search" parameter, which results in this slightly adjusted bing-redirect:
- old:
{search/^\?.*&?q=(.+?)(&.*|#.*|$)/https://www.ixquick.com/do/dsearch?query=$1&cat=web&pl=opensearch&language=english} - new:
{search/(^\?|.*&)q=(.+?)(&.*|$)/https://www.ixquick.com/do/dsearch?query=$2&cat=web&pl=opensearch&language=english} - alternative format:
https://www.ixquick.com/do/dsearch?query={search/(^\?|.*&)q=(.+?)(&.*|$)/$2}&cat=web&pl=opensearch&language=english
... pick whichever format you prefer.
edit: fuck! the ^\?.*&?q= part is flawed, for example: ?testq=wrong without a q=right, however it works for ?testq=disney&q=porn but not for ?q=porn&testq=disney, ie if you want porn instead of disney :)
got it! (^\?|.*&)q=
edit2:
for Redirector:
https?://www.bing.com/search(\?|.*&)q=(.+?)(&.*|#.*|$) - this one "needs" the anchor part again!
Redirect to: https://www.ixquick.com/do/dsearch?query=$2&cat=web&pl=opensearch&language=english
please try to break this one y'all ;)
earthlng
on 24 Jun 2017
We need some really generic ones, but I have no idea how bad the breakage could get.
I would stay away from generic ones if you care about efficiency and no breakage.
Does Request Control offer anything that Redirector doesn't or vice versa?
Redirector
- pros
- example result
- control over processing -> "Process matches"
- very detailed console output
- simpler terms (iframe instead of embedded document)
- import/export rules
- Exclude Pattern (you can probably achieve the same in RC with a bit of regex-fu)
- custom description
- cons
- very detailed console output (performance impact, log "pollution", no pref to disable it?)
- simpler terms ("other" instead of CSP report (which I guess is part of "other"), etc)
- no direct access to only parts of the url (ie RC's "parameters")
- doesn't support multiple domains/paths in one rule
- no "TLD's" support (fe. see default google rule in RC)
- no "block" request
- no indicator in the UI for redirects
Redirector's author said the addon is in "maintenance mode" and he won't add new features.
RC's author still seems to be very actively maintaining the addon and is open to add new features.
RC does some "magic" in the background which is great (and seems to work) but not very transparent.
fe. it strips additional parameters (&xyz=foo&bar=etc) with "Filter" rules and does urldecoding if necessary. I'd personally prefer a bit more control over this, especially the encoding/decoding stuff.
Maybe we can lobby him/her to add that (with a default "auto" option for the current behavior)
edit: Redirector allows/requires to order the rules but idk how useful that is. It probably makes it more complicated if you have a lot of rules
earthlng
on 24 Jun 2017
This appears to be breaking some internal Amazon links (such as "Warehouse Deals" for example),
presumably it`s the ref= part.
With regard to this, how exactly are these "2" filter rules applied ?
Separately ?, must they be in a specific order ?
This stuff gives me a headache at the best of times - intriguing stuff though !
GitCurious
on 25 Jun 2017
OT: @Thorin-Oakenpants Good to know, looks like im not John Cena!
Forsaked
on 26 Jun 2017
FYI atm there's a bug that a redirected Document still gets added to the history. However the request doesn't touch the network. (that was fixed in 54 and also backported to ESR here)
re: https://www.reddit.com/r/firefox/comments/6j6w0v/how_to_automatically_clean_up_urls_by_removing/djc7w8v/
earthlng
on 26 Jun 2017
The *.jpg?*, *.png?* ... image filters break images on www.kickstarter.com.
southwindcg
on 27 Jun 2017
Try whitelisting https://ksr-ugc.imgix.net/
Atavic
on 27 Jun 2017
Will keep eye on this addon, since it has a lot of potential, but will hold until case insesitive isn't sorted out.
crssi
on 30 Jun 2017
@GitCurious REF is tricky and WILL make breakage.
There are no orders in RC.
The first rule takes the last URL in the URL (damn if this makes sense :))
And the second skips this action when you hit some login pages.
crssi
on 30 Jun 2017
There are also some other issues. I have placed an Issue on RC github.
crssi
on 30 Jun 2017
Sure you can. Its just a JSON.
This is the same:
[{"pattern":{"scheme":"*","host":["www.imdb.com"],"path":["title/*","name/*","character/*"]},"types":["main_frame"],"action":"filter","active":true,"paramsFilter":{"values":["ref_"],"pattern":"ref_"},"skipRedirectionFilter":true}]
crssi
on 11 Jul 2017
Damn cool... how do you do that?
crssi
on 11 Jul 2017
Ah, ok, I see now... thx :)
crssi
on 11 Jul 2017
@crssi Google has &biw= and others, see RemoveGoogleTracking.
Atavic
on 4 Aug 2017
Thank you @Atavic. Will take a look into in the next days
crssi
on 8 Aug 2017
For redirect skipping RC is no match to Skip Redirect (WE), especially now where author implemented same domain detection... see https://github.com/sblask/webextension-skip-redirect/issues/30
I will not try to mimic the same functionality on RC anymore.
For basic tracking removal the Link Cleaner does a nice job, but RC can be used as a supplement for more advanced rules or for additional tracking options removal that are not covered in LC.
crssi
on 8 Aug 2017
@ghacksuserjs from RemoveGoogleTracking:
'biw', // offsetWidth
'bih', // offsetHeight
Apparently related to screen fingerprint.
Atavic
on 8 Aug 2017
@Atavic
To to be sure... should I add to https://github.com/ghacksuserjs/ghacks-user.js/issues/149#issuecomment-310173763 the following:
biw, bih, ei, sa, ved, source, prmd, bvm, bav, psi, stick, dq, ech, gs_gbg, gs_rn, cp, scroll, vet, yv, ijn, iact, forward, ndsp, csi, tbnid, pbx, dpr, pf, gs_rn, gs_mss, pq, cp, oq, sclient, gs_l, aqs, psi
Observation: From what I see on the source code of RemoveGoogleTracking, its more complex than just to remove those parameters.
crssi
on 9 Aug 2017
I have redesigned some rules and dropped some from the scratch and have issued few issues to @tumpio... specially this one.
When some more is known about the issue I will close this topic and open a new topic (referenced to this one) and post from scratch.
crssi
on 18 Aug 2017
@Thorin-Oakenpants
If you agree I would like to close this topic, since @tumpio is not active almost 2 months and there is better approach to deal with skipping redirect and url cleaning.
I can open new "post" with all together solution with setup I am using for skip redirect and url cleaning in the next few days?
crssi
on 29 Sep 2017
example to skip youtu.be:
pattern: scheme: http/https host: youtu.be path: *
types: Document
action: Redirect
redirect to: https://www.youtube.com/watch?v={pathname:1}
example url: https://youtu.be/nqbUkThGlCo
Request Control supports direct access to named parameters
The named parameter pathname in the example link is /nqbUkThGlCo and {pathname:1} strips the slash using substring extraction
earthlng
on 14 Oct 2017
My love to all of you
securingmom
on 5 Jan 2021
Related issues
Thorin-Oakenpants
路
5Comments
Thorin-Oakenpants
路
3Comments
grauenwolfe
路
7Comments
Thorin-Oakenpants
路
4Comments
TerkiKerel
路
4Comments
Most helpful comment
example to skip youtu.be:
pattern: scheme:
http/httpshost:youtu.bepath:*types:
Documentaction:
Redirectredirect to:
https://www.youtube.com/watch?v={pathname:1}example url: https://youtu.be/nqbUkThGlCo
Request Control supports direct access to named parameters
The named parameter pathname in the example link is
/nqbUkThGlCoand{pathname:1}strips the slash using substring extraction