µMatrix thread

General discussion about web technology.
Post Reply
barbaz
Senior Member
Posts: 10834
Joined: Sat Aug 03, 2013 5:45 pm

µMatrix thread

Post by barbaz »

Since µMatrix has been getting some attention here recently from a number of posters, I think it's time for a dedicated thread here about it, and explaining what it is and isn't from a user's standpoint, as well as some basics how-to. So I tried µMatrix dev build in a new profile in Firefox 37.0.2 alongside NoScript (default NS configuration), just to get a feel for what it is and how to use it.

Some NoScript users may want to use µMatrix alongside NoScript, so some information here can't hurt.

Basically, µMatrix is an advanced request filter and "browser firewall". It works in two concurrent ways:
1) Outright blocks all domains listed in a HOSTS file.
2) Allow or block requests based on a series of rules of the form

Code: Select all

matrix-off: [site] true
[source|*] [destination|*] [type|*] [allow|block]
type can also be cookies, which doesn't affect requests but controls whether cookies can be set.
If type is script, it should block inline scripts too.

µMatrix also has a few miscellaneous privacy options. One of these is the option to spoof UA's and change UA every so often, but that one seems to rely on an injected <script> element...
I have not played with the referer spoofing nor the Strict HTTPS (forbid mixed content) feature.


Parts of the interface are somewhat difficult to figure out... and the documentation is either lacking or hard to find, so here is what I figured out about it that I don't think is necessarily self-explanatory or easy to discover.

Pop-up:
- In the pop-up, there are a bunch of red and green cells. Red = Blocked, Green = Allowed, Light Red = Blocked (inherited), Light Green = Allowed (inherited). Click the top half of the light-colored cells to explicitly set an Allowed status, and the bottom-half to explicitly set a Deny status. Otherwise, just click the cell to toggle that permission status.
- Click the cell in the very top left (either blue or black) to set the scope of the rules being edited in the popup.
- Once you have made the changes you want, click the lock to save them. If you UNblocked anything, you need to use µMatrix's own reload page button in its panel, otherwise the "blocked" state will be cached on reload.

Dashboard: (to access, go to the +, then "Go to dashboard")
Mostly pretty self explanatory, except for the "My rules" tab. There, you see "Permanent rules" on the left and "Temporary rules" on the right. The way it works is you modify the "Temporary rules" then you "Commit" to save the changes and update "Permanent Rules".
To actually make modifications: click on a single entry in Temporary rules to mark it for deletion; or, click "Edit" and manually type rules.

If you use µMatrix alongside NoScript, you might want to add the following rule:

Code: Select all

* * script allow
to let NoScript manage script permissions.
(NoScript uses its script permission status for more than just whether or not to run scripts. Super power users may want to use both tools to manage script permissions.)


I'm not an experienced µMatrix user, so if I got something wrong, please let me know.


Hope this is helpful.
*Always* check the changelogs BEFORE updating that important software!
-
User avatar
lakrsrool
Senior Member
Posts: 195
Joined: Wed Nov 12, 2014 4:20 pm

Re: µMatrix thread

Post by lakrsrool »

Something that is really bothersome with Umatrix, that is some websites are completely blocked by Umatrix. Some of these are "finance" website links off of Yahoo finance that I've been using for years.

Here is an example: http://us.rd.yahoo.com/finance/external ... _ven=YAHOO

Just messing around, I'm able to get the website to be allowed (not completely work however) by using this link: http://www.thestreet.com/story/13275111 ... _ven=YAHOO

The page using the top link (what a user gets if clicking on the link in Yahoo) is not loaded at all and I get the message: "uMatrix has prevented the following page from loading" (referring to the above page).

There is no way to allow anything because the page is blocked outright thus nothing provided to the user to "adjust".

How does one prevent this from happening?

Why is Umatrix blocking the site (I know, presumably for security reasons, but as I said I've been linking to this source "www.thestreet.com" off of Yahoo for the past 10+ years with no problems)?

One would assume that Umatix apparently assumes this website to be extremely high risk. Well I would have to say I find that odd considering how much I've used the web site through the years.

I don't want to reduce the security globally that Umatrix provides of course, but it would be nice if Umatrix would provide more information on this instead of just the message I posted above and it would also be more "friendly" to also provide ways to allow the website.

Also I might be wrong about this, but it seems like some of these links are eventually not blocked later on (but I can't currently verify that as of now).

Afterthought: I like the way Umatrix works, much more intuitive and easier than Policeman by a wide margin, but I have to say I find myself constantly adjusting websites that I've used for literally decades and are essential to me, just one of many examples: Schwab web pages that I would have to presume safe and I simply have to use which Umatrix blocks at many levels. It makes one wonder why anyone would even use the internet at all the way Umatrix blocks so much of a website which would suggest to the user all of what is blocked is for a reason. Well anyway I digress.....

Thanks for any help in advance on what to do about Umatrix totally blocked websites.
Last edited by lakrsrool on Fri Sep 11, 2015 3:59 pm, edited 2 times in total.
Mozilla/5.0 (Windows NT 6.1; WOW64; rv:40.0) Gecko/20100101 Firefox/40.0
barbaz
Senior Member
Posts: 10834
Joined: Sat Aug 03, 2013 5:45 pm

Re: µMatrix thread

Post by barbaz »

Check if it got listed in one of µMatrix's HOSTS files or other auto-downloaded blacklists?
*Always* check the changelogs BEFORE updating that important software!
-
User avatar
lakrsrool
Senior Member
Posts: 195
Joined: Wed Nov 12, 2014 4:20 pm

Re: µMatrix thread

Post by lakrsrool »

barbaz wrote:Check if it got listed in one of µMatrix's HOSTS files or other auto-downloaded blacklists?
Thanks for the quick reply.

In regards to just "HOSTS" files: According to Umatrix there are 75,729 distinct blocked hostnames within 7 different services. So are you saying I'd have to search each of the 7 services to find this. And if so what do I use as the "search" criteria?

One a side note: One wonders if this is all worth it for security reasons.

I never could get the video to work on that web site (using the link I got to load) even though I allowed ALL permissions in NoScript and completely disabled Umatrix for that pages scope and still can't get the video to work like it will in my IE browser without any NoScript, Umatrix.

I find I have to have IE available to test to see what a website is supposed to do with all of the security levels I've got on the Firefox browser I usually use. It really does become cumbersome to have to use all of this security and would sure be nice to just use the internet the way it was intended.
Mozilla/5.0 (Windows NT 6.1; WOW64; rv:40.0) Gecko/20100101 Firefox/40.0
User avatar
lakrsrool
Senior Member
Posts: 195
Joined: Wed Nov 12, 2014 4:20 pm

Re: µMatrix thread

Post by lakrsrool »

OK I took "us.rd.yahoo.com" figuring removing this part of the URL allowed the website to work in Umatrix and searched all 7 of the Host service blockers and did not find this value in any of them.

I am not aware of any "blacklists" of any kind, I know I haven't set them and can't find any reference to this in Umatrix.

If you're referring to what Umatrix says at the bottom of their interface well it says there are 23 blocklisted domains, but that doesn't help me much because I have no way of knowing what domains their referring to and not that I'd necessarily want to, but the user can't do anything about this through the Umatrix interface anyway.

It is apparent that Umatrix needs a lot more "Help" contents made available to users (like is found in NoScript for example), it is far too esoteric in the manner in which it works. Do you know of any user help files provided by Umatrix to users?
Last edited by lakrsrool on Fri Sep 11, 2015 1:12 am, edited 1 time in total.
Mozilla/5.0 (Windows NT 6.1; WOW64; rv:40.0) Gecko/20100101 Firefox/40.0
User avatar
lakrsrool
Senior Member
Posts: 195
Joined: Wed Nov 12, 2014 4:20 pm

Re: µMatrix thread

Post by lakrsrool »

Off of the topic on how to make uMatrix allow the website to even load (stop Umatrix from blocking the entire website) I'm finally able to get the video to work (this is with Umatrix scope disabled). I had to continue to enable more and more sites as they showed up in Noscript each page reload that probably ultimately totaled easily 45+ sites (had to scroll quite a ways to see full list) and also had to keep Umatrix disabled for that scope and I disabled all of what Disconnect blocks as well as what Avast web security blocks and it finally took my disabling the add-on Ublock altogether to get the video to work. So I understand the concept is to only allow the bare minimum that is required to make a website work, but as you can see the possible combinations to know what the actual "required" sites are to do this it would be something like 45 to the 45th power I suppose, in other words a huge number that would take me literally many weeks of 12 hour days to determine and that is with keeping the Ublock add-on disabled and Umatrix disabled for that websites scope. To then figure out what needs to be allowed in Umatrix and then figure out where Ublock is causing a problem clearly makes trying to make a website safe and allow only what is necessary to have it work properly is not just bordering insanity but literally is insanity considering the effort needed to do so.

I am understanding what the concept is, right? That is to only allow what is really necessary for a website to work in order to provide as much security as possible? And of course virtually all of the sites that are part of making up a website have a myriad of opinions as to whether they are safe or potentially malicious at some level which seems to depend upon personal "subject" opinion more than anything. It does in many ways when it comes to websites like this to be an extremely nebulous endeavor to find an optimum level of security in these cases.
Mozilla/5.0 (Windows NT 6.1; WOW64; rv:40.0) Gecko/20100101 Firefox/40.0
User avatar
lakrsrool
Senior Member
Posts: 195
Joined: Wed Nov 12, 2014 4:20 pm

Re: µMatrix thread

Post by lakrsrool »

Well anyway I digress once again, let me know when you find out how to make Umatrix allow a website like the one I've posted: http://us.rd.yahoo.com/finance/external ... _ven=YAHOO

I run into this from time to time and in many cases it's a website that I've been using or sometimes decades to do financial research which worked just fine and apparently gave me no problems until I installed the Umatrix add-on. (although in this specific case it seems NoScript was also involved which I've been using for years).
Mozilla/5.0 (Windows NT 6.1; WOW64; rv:40.0) Gecko/20100101 Firefox/40.0
gorhill
Junior Member
Posts: 48
Joined: Sun Mar 30, 2014 12:19 pm

Re: µMatrix thread

Post by gorhill »

"rd.yahoo.com" is listed in hpHosts.

To override the block, it's just a matter of bringing up the matrix while on the warning page, and override the block rule by just clicking on it.

Image

Force a reload from the matrix will cause the blocked page to reload with the new override rule. Click the padlock to make the override permanent.

If you want the override to apply everywhere, not just "yahoo.com" , don't forget to select the global scope before creating the override.
Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:43.0) Gecko/20100101 Firefox/43.0
User avatar
lakrsrool
Senior Member
Posts: 195
Joined: Wed Nov 12, 2014 4:20 pm

Re: µMatrix thread

Post by lakrsrool »

I apparently wasn't able to find it in hpHosts because I was looking for "us.rd.yahoo.com" instead of "rd.yahoo.com".

Is clicking on the little red triangle in upper left corner of the site make it a "global" change? (it would be nice if all of this were documented somewhere by Umatrix)

I was just curious about how to set to "global" because on the point of a "global" change in this case I wouldn't want it "global" I presume because the "thestreet.com" website itself is not what is blocked but instead the website gets blocked because it originates (is redirected) from "Yahoo finance" (that being "http://us.rd.yahoo.com/finance/...") so I'd only want to unblock what gets "inherited" from "Yahoo finance" in this case. At least that is the way I see it.

What I found a bit peculiar is that if I lock the change prior to reloading then the change is made "permanent" in "my rules". But if I do the Umatrix "reload" as described above in the previous post then the lock icon gets "closed" (usually a "reload" does not "close" the lock image) which surprises me and surprises me even more so because I find that the change is NOT "permanent" in "my rules" (which the closed lock would suggest is the case) therefore requiring me to go to "My rules" and press "Commit" to move the "Temporary rule" to become a "Permanent rule". So it appears from what I can see it is best to lock the change if one wants to make the rule "permanent" (without having to go to "my rules" to do so) as opposed to forcing a "reload" since after the reload one would think the rule was "permanent" (because the lock is closed) when it really is not. This makes me wonder if the lock closing after a forced reload on it's own is a "bug" since that's not how it usually works and is not the expectation but rather all a reload should do is reload (repaint the page) according the the "temporary" settings that would typically still require the user to click on the lock to make it permanent. In this case the reload made it "look" permanent because the lock is closed yet it is not permanent because it remains a "Temporary Rule" only in place until the browser is closed because the "Temporary Rule" will not persist for the next time the browser is started again. So if a forced reload is done the user will have to go to "My rules" to make it permanent because of the fact the lock is closed at that point and therefore the user CANNOT lock it using the Umatrix icon interface thus requiring the user to go to "My rules" to have the rule made "permanent" which is why again as I said it seems like a "bug" to me to actually "close" the "lock" when the forced reload is done in this case.
Mozilla/5.0 (Windows NT 6.1; WOW64; rv:40.0) Gecko/20100101 Firefox/40.0
User avatar
lakrsrool
Senior Member
Posts: 195
Joined: Wed Nov 12, 2014 4:20 pm

Re: µMatrix thread

Post by lakrsrool »

After getting Umatrix to not block this Thestreet.com website I found that the website did not work, more specifically the video. Now I had to get the website to work with NoScript, Umatrix, Ublock etc. So to first isolate just NoScript I disabled everything other than NoScript and literally spend many more hours first getting NoScript to allow the video to work with the minimum number of sites required that I could determine and then enabled Umatrix and worked with Umatrix to do the same. I found that if I would "revert all temporary changes" in Umatrix I would lose some settings that were previously available and I had to retrace all my steps once again with NoScript to get back to where I was with Umatrix again so I went through 3 cycles between these two add-ons. I then enabled Ublock and found that Ublock was also blocking the video to my surprise. So I had to check the Ublock logger to eventually find out there is a javascript entry in EasyPrivacy (kgoogleanalytics.js) that is being blocked that causes the video to fail. So I had to disable EasyPrivacy in Ublock to get the video to work and I also notified the EasyPrivacy forum (actually EasyList forum) to let them know about this. Btw, NoScript and Umatrix was blocking a lot that kept me from registering with EasyList and in fact it took about 15 minutes to finally get the captcha to work to get to open an account with Easylist and find their verification link in my SPAM email folder and now I have to wait to have my post verified. After getting this all done I found that ABP had to remain disabled for this page and Disconnect for all Google sites had to be disabled as well.

All of this that took probably a total of 8+ hours just to get a website to work with NoScript, Umatrix and Ublock (and disabling ABP and Google sites in Disconnect). :roll:

One wonders how important it is in the first place for all of these sites to be blocked that actually needed to be "allowed" when the fact is the website I've been using for many years in the past without any of these add-ons won't work without all these sites being allowed in all these add-ons. :?: It is certainly cumbersome to try and stay secure so much so I thought I'd post all of what I had to do to just get this one website to finally work correctly. :idea: (I'm not complaining about the add-ons so much as I'm just pointing out how inconvenient it can be just to avoid those deviants who's goal is to anonymously do cyber damage.) :twisted:
Mozilla/5.0 (Windows NT 6.1; WOW64; rv:40.0) Gecko/20100101 Firefox/40.0
gorhill
Junior Member
Posts: 48
Joined: Sun Mar 30, 2014 12:19 pm

Re: µMatrix thread

Post by gorhill »

lakrsrool wrote:it would be nice if all of this were documented somewhere by Umatrix
There is a wiki -- ("Documentation" link in the "About" pane). I warn that uMatrix is for advanced users, and I completely expect that advanced users would find their way to the doc.
Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:43.0) Gecko/20100101 Firefox/43.0
barbaz
Senior Member
Posts: 10834
Joined: Sat Aug 03, 2013 5:45 pm

Re: µMatrix thread

Post by barbaz »

That link to the wiki isn't working for me, so in case it's not working for others either: https://github.com/gorhill/uMatrix/wiki

@gorhill: If you would like to copy parts of my OP to the µMatrix wiki, feel free.
*Always* check the changelogs BEFORE updating that important software!
-
barbaz
Senior Member
Posts: 10834
Joined: Sat Aug 03, 2013 5:45 pm

Re: µMatrix thread

Post by barbaz »

I've just noticed that µMatrix has become available for SeaMonkey (thanks gorhill for making it available beyond Firefox). Installed to my browser.

Also, I had a bit of a hard time finding documentation about the "cloud storage support" feature, so to save others some time: https://github.com/gorhill/uBlock/wiki/Cloud-storage
In summary, it's the option to sync custom µMatrix rules via the browser's built-in Sync implementation.

I also have a question, how to disable the placeholders? They can make µMatrix detectable by web pages.
EDIT Can partially disable by checking "Collapse placeholder of blocked elements", but this leaves behind a "display:none ! important;" on the element. At least this way it can't be detected as µMatrix.
EDIT2 I can't find a setting for this either in about:config nor inside the addon's sqlite file :(
EDIT3 Now it's mysteriously working? :?
EDIT4 Ah, got it (partially). Make all extensions.umatrix.placeholder* about:config prefs to blank user-set string, then restart browser. Except that it's setting the placeholder document back... Image EDIT5 However that's not webpage-detectable AFAIK, all attempts to tryr are coming back with "Permission denied". So I guess this quirk not matter much.
*Always* check the changelogs BEFORE updating that important software!
-
barbaz
Senior Member
Posts: 10834
Joined: Sat Aug 03, 2013 5:45 pm

Re: µMatrix thread

Post by barbaz »

Some more things I've found about it:
- The order of "My rules" seems immaterial. The only way I've found to figure out what's taking precedence over what, is to look at the matrix.
- To go to the dashboard from the panel (at least in SeaMonkey), click the "uMatrix 0.9.3.1" bar at the top.
*Always* check the changelogs BEFORE updating that important software!
-
User avatar
Thrawn
Master Bug Buster
Posts: 3106
Joined: Mon Jan 16, 2012 3:46 am
Location: Australia
Contact:

Re: µMatrix thread

Post by Thrawn »

barbaz wrote:(thanks gorhill for making it available beyond Firefox).
I believe he supports Pale Moon, too.
======
Thrawn
------------
Religion is not the opium of the masses. Daily life is the opium of the masses.

True religion, which dares to acknowledge death and challenge the way we live, is an attempt to wake up.
Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:42.0) Gecko/20100101 Firefox/42.0
Post Reply