[FIXED]ABE incorrectly thinks most requests are type "other"

Bug reports and enhancement requests
barbaz
Senior Member
Posts: 10841
Joined: Sat Aug 03, 2013 5:45 pm

[FIXED]ABE incorrectly thinks most requests are type "other"

Post by barbaz »

Thanks PLD for discovering this.

Firefox 54.0
NoScript 5.0.7.1rc1
new profile

STR:
1) NoScript Options > Advanced > ABE > USER, add

Code: Select all

Site ALL
Deny INC(OTHER)
2) try to browse to https://noscript.net/

Expected results: site loads and looks normal

Actual results: site is totally blocked by ABE. Related console messages -

Code: Select all

[ABE] < ALL> Deny INCLUSION(OTHER) on {GET https://noscript.net/ <<< chrome://browser/content/browser.xul - 1}
USER rule:
Site ALL
Deny INCLUSION(OTHER)
So I tried changing the ABE rule to -

Code: Select all

Site ALL
Accept from chrome: moz-nullprincipal:
Deny INC(OTHER)
While the NoScript page now loads, it is quite badly messed up. New related console messages -

Code: Select all

[ABE] < ALL> Deny INCLUSION(OTHER) on {GET https://noscript.net/oss.css <<< https://noscript.net/ - 1}
USER rule:
Site ALL
Accept from chrome: moz-nullprincipal:
Deny INCLUSION(OTHER)
[ABE] < ALL> Deny INCLUSION(OTHER) on {GET https://noscript.net/iasw.jpg <<< https://noscript.net/ - 1}
USER rule:
Site ALL
Accept from chrome: moz-nullprincipal:
Deny INCLUSION(OTHER)
[ABE] < ALL> Deny INCLUSION(OTHER) on {GET https://api.flattr.com/button/flattr-badge-large.png <<< https://noscript.net/ - 1}
USER rule:
Site ALL
Accept from chrome: moz-nullprincipal:
Deny INCLUSION(OTHER)
[ABE] < ALL> Deny INCLUSION(OTHER) on {GET https://noscript.net/noscript/logo.png <<< https://noscript.net/ - 1}
USER rule:
Site ALL
Accept from chrome: moz-nullprincipal:
Deny INCLUSION(OTHER)
[ABE] < ALL> Deny INCLUSION(OTHER) on {GET https://noscript.net/noscript/ss0.png <<< https://noscript.net/ - 1}
USER rule:
Site ALL
Accept from chrome: moz-nullprincipal:
Deny INCLUSION(OTHER)
[ABE] < ALL> Deny INCLUSION(OTHER) on {GET https://noscript.net//images/sponsored-by-nlnet.png <<< https://noscript.net/ - 1}
USER rule:
Site ALL
Accept from chrome: moz-nullprincipal:
Deny INCLUSION(OTHER)
[ABE] < ALL> Deny INCLUSION(OTHER) on {GET https://noscript.net/wc06.jpg <<< https://noscript.net/ - 1}
USER rule:
Site ALL
Accept from chrome: moz-nullprincipal:
Deny INCLUSION(OTHER)
[ABE] < ALL> Deny INCLUSION(OTHER) on {GET https://noscript.net//badge-flashgot.png <<< https://noscript.net/ - 1}
USER rule:
Site ALL
Accept from chrome: moz-nullprincipal:
Deny INCLUSION(OTHER)
[ABE] < ALL> Deny INCLUSION(OTHER) on {GET https://noscript.net/badge-easyspeedy.png <<< https://noscript.net/ - 1}
USER rule:
Site ALL
Accept from chrome: moz-nullprincipal:
Deny INCLUSION(OTHER)
[ABE] < ALL> Deny INCLUSION(OTHER) on {GET https://noscript.net/donate2.gif <<< https://noscript.net/ - 1}
USER rule:
Site ALL
Accept from chrome: moz-nullprincipal:
Deny INCLUSION(OTHER)
[ABE] < ALL> Deny INCLUSION(OTHER) on {GET https://noscript.net/noscript.ico <<< https://noscript.net/ - 1}
USER rule:
Site ALL
Accept from chrome: moz-nullprincipal:
Deny INCLUSION(OTHER)
Disabling e10s seems to get things back to normal. Console output for comparison -

Code: Select all

[ABE] < ALL> Deny INCLUSION(OTHER) on {GET https://noscript.net/noscript.ico <<< https://noscript.net/ - 1}
USER rule:
Site ALL
Accept from chrome: moz-nullprincipal:
Deny INCLUSION(OTHER)
*Always* check the changelogs BEFORE updating that important software!
-
User avatar
Giorgio Maone
Site Admin
Posts: 9454
Joined: Wed Mar 18, 2009 11:22 pm
Location: Palermo - Italy
Contact:

Re: [e10s] ABE incorrectly thinks most requests are type "ot

Post by Giorgio Maone »

Please check latest development build 5.0.8rc2, thanks.
Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:54.0) Gecko/20100101 Firefox/54.0
barbaz
Senior Member
Posts: 10841
Joined: Sat Aug 03, 2013 5:45 pm

Re: [e10s] ABE incorrectly thinks most requests are type "ot

Post by barbaz »

Much better, but now Deny INC(OTHER) no longer blocks Websocket connections. This both with and without e10s enabled.

Test page: https://www.websocket.org/echo.html

EDIT
Here's the console message produced for plain Deny INC, NoScript 5.0.8rc2, SeaMonkey -

Code: Select all

[ABE] < echo.websocket.org> Deny INCLUSION on {GET http://echo.websocket.org/?encoding=text <<< http://www.websocket.org/echo.html - 16}
TEST rule:
Site echo.websocket.org
Deny INCLUSION
Console output from 5.0.8rc1, same ABE rule -

Code: Select all

[ABE] < echo.websocket.org> Deny INCLUSION on {GET http://echo.websocket.org/?encoding=text <<< http://echo.websocket.org/?encoding=text, http://www.websocket.org/echo.html - 1}
TEST rule:
Site echo.websocket.org
Deny INCLUSION
*Always* check the changelogs BEFORE updating that important software!
-
PLD
Junior Member
Posts: 30
Joined: Fri May 08, 2015 7:00 am

Re: [e10s] ABE incorrectly thinks most requests are type "ot

Post by PLD »

@barbaz: Thank you for recognizing the INC issue and starting this thread.
@giorgio: Thanks for 5.08rc2
barbaz wrote:Much better, but now Deny INC(OTHER) no longer blocks Websocket connections. This both with and without e10s enabled.
That would be a step in the right direction given that it is its own type:

Chrome webRequest.ResourceType:

Code: Select all

"main_frame", "sub_frame", "stylesheet", "script", "image", "font", "object", "xmlhttprequest", "ping", "csp_report", "media", "websocket", or "other"
Firefox webRequest.ResourceType:

Code: Select all

"main_frame", "sub_frame", "stylesheet", "script", "image", "object", "object_subrequest", "xmlhttprequest", "xbl", "xslt", "ping", "beacon", "xml_dtd", "font", "media", "websocket", "csp_report", "imageset", "web_manifest", "other"
But ABE would have to recognize it. I did try Deny INC(WEBSOCKET) and the latest rc didn't accept it.

I'm still testing 5.0.8rc2 INC handling and have two questions. Given this:

Code: Select all

Site .td.test
Deny INC(SCRIPT)
Q1: If a page includes an iframe from test.td.test and that iframe has inline script, should that script execute?
Q2: If a page includes an iframe from test.td.test and that iframe includes script from some.other.domain (that is allowed), should that script execute?

I'm thinking that Deny INC(SCRIPT) is not supposed to cascade to those items, but would like to confirm. Thanks.
Mozilla/5.0 (Windows NT 6.1; WOW64; rv:52.0) Gecko/20100101 Firefox/52.0
User avatar
Giorgio Maone
Site Admin
Posts: 9454
Joined: Wed Mar 18, 2009 11:22 pm
Location: Palermo - Italy
Contact:

Re: [e10s] ABE incorrectly thinks most requests are type "ot

Post by Giorgio Maone »

PLD wrote:@barbaz: Thank you for recognizing the INC issue and starting this thread.
@giorgio: Thanks for 5.08rc2
barbaz wrote:Much better, but now Deny INC(OTHER) no longer blocks Websocket connections. This both with and without e10s enabled.
That would be a step in the right direction given that it is its own type:
It's actually a problem of balancing backward compatibility (some types are relatively recent) and usefulness in current Firefox versions.

Therefore in RC3 I'm gonna update the current set of recognized INCLUDE(XYZ), with some convenience/compat aliases, as follows (where not otherwise specified in parentheses, the name maps directly to the nsIContentPolicy.TYPE_$name constant):

Code: Select all

BEACON; CSP_REPORT; CSS (TYPE_STYLESHEET); DATAREQUEST; DOCUMENT; DTD; FETCH; FONT; IMAGE (TYPE_IMAGE, TYPE_IMAGESET); IMAGESET; INVALID; MEDIA; OBJ (TYPE_OBJECT, TYPE_OBJECT_SUBREQUEST); OBJECT; OBJECT_SUBREQUEST; OBJSUB (TYPE_OBJECT_SUBREQUEST); OTHER (TYPE_OTHER, TYPE_WEBSOCKET); PING; REFRESH; SCRIPT; STYLESHEET; SUBDOC (TYPE_SUBDOCUMENT); SUBDOCUMENT; UNKNOWN (TYPE_OTHER); WEBSOCKET; WEB_MANIFEST; XBL; XHR (TYPE_XMLHTTPREQUEST, TYPE_DATAREQUEST); XMLHTTPREQUEST; XSLT
As you can see, "OTHER" matches TYPE_OTHER (anything not yet uniquely labeled in nsIContentPolicy) and TYPE_WEBSOCKET for backward compatibility. If you don't want to match TYPE_WEBSOCKET but just TYPE_OTHER, use INC(UNKNOWN). If you want to match TYPE_WEBSOCKET only, use INC(WEBSOCKET).
PLD wrote:

Code: Select all

Site .td.test
Deny INC(SCRIPT)
Q1: If a page includes an iframe from test.td.test and that iframe has inline script, should that script execute?
Nope, INCLUDE doesn't match inline content, just discrete loads (like those triggered by <script src="xyz"></script> elements).
PLD wrote: Q2: If a page includes an iframe from test.td.test and that iframe includes script from some.other.domain (that is allowed), should that script execute?
Nope, see above.
Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:54.0) Gecko/20100101 Firefox/54.0
User avatar
Giorgio Maone
Site Admin
Posts: 9454
Joined: Wed Mar 18, 2009 11:22 pm
Location: Palermo - Italy
Contact:

Re: [e10s] ABE incorrectly thinks most requests are type "ot

Post by Giorgio Maone »

Please check latest development build 5.0.8rc3, thanks.
Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:54.0) Gecko/20100101 Firefox/54.0
PLD
Junior Member
Posts: 30
Joined: Fri May 08, 2015 7:00 am

Re: [e10s] ABE incorrectly thinks most requests are type "ot

Post by PLD »

Giorgio Maone wrote:
PLD wrote:Therefore in RC3 I'm gonna update the current set of recognized INCLUDE(XYZ), with some convenience/compat aliases <snip>
That is great news! A neat way of handling backward compat too. Looks like the bases are covered except for one thing: Would anyone want to address TYPE_IMAGE without also addressing TYPE_IMAGESET? I haven't thought about it and don't know the answer.

As for my rc2 testing, so far so good except:

Code: Select all

Site ALL
Deny INC(SCRIPT)
Is not blocking an external script file which is included via src scheme of ftp: or file: (the file: schemed inclusion happening when test page is loaded via file: - a page relative load actually). Same applies to a few other INC types I tried. I also tried:

Code: Select all

Site ALL
Deny ALL
Test page loads from file: and both file: and ftp: schemed script/other inclusions happen. I'm aware of Thrawn's 2015 comment about ftp being out of scope, but.

Edited to clarify and now see 5.0.8rc3...
Mozilla/5.0 (Windows NT 6.1; WOW64; rv:52.0) Gecko/20100101 Firefox/52.0
barbaz
Senior Member
Posts: 10841
Joined: Sat Aug 03, 2013 5:45 pm

Re: [e10s] ABE incorrectly thinks most requests are type "ot

Post by barbaz »

Giorgio Maone wrote:Please check latest development build 5.0.8rc3, thanks.
Nice, thanks! :D

Couple questions -

1) Did OTHER include more types before, specifically BEACON, CSP_REPORT, and WEB_MANIFEST? Not that I've used it for such purpose, just wondering from backwards compatible point of view.

2) Should TYPE_FETCH be included by XHR?
*Always* check the changelogs BEFORE updating that important software!
-
PLD
Junior Member
Posts: 30
Joined: Fri May 08, 2015 7:00 am

Re: [e10s] ABE incorrectly thinks most requests are type "ot

Post by PLD »

Saw this in 5.0.8rc3:

Code: Select all

HTTPVERB  : 'GET' | 'POST' | 'PUT' | 'PATCH' | 'DELETE'; (ABE.g line 46)
Tried adding ABE rules with HEAD, TRACE, and OPTIONS (not accepted).

New abe_rules.pdf says:
1.2 Methods
The <method> component of a <predicate> can be any HTTP method (GET, POST, HEAD, PUT, DELETE, TRACE, OPTIONS) with the addition of 3 “pseudo” methods:
MDN lists:
GET, HEAD, POST, PUT, DELETE, CONNECT, OPTIONS, TRACE, PATCH
Mozilla/5.0 (Windows NT 6.1; WOW64; rv:52.0) Gecko/20100101 Firefox/52.0
User avatar
Giorgio Maone
Site Admin
Posts: 9454
Joined: Wed Mar 18, 2009 11:22 pm
Location: Palermo - Italy
Contact:

Re: [e10s] ABE incorrectly thinks most requests are type "ot

Post by Giorgio Maone »

That's a grammar regression I had already fixed yesterday, but it's so minor that I didn't deem worth of a RC4 just for that ;)
Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:54.0) Gecko/20100101 Firefox/54.0
User avatar
Giorgio Maone
Site Admin
Posts: 9454
Joined: Wed Mar 18, 2009 11:22 pm
Location: Palermo - Italy
Contact:

Re: [e10s] ABE incorrectly thinks most requests are type "ot

Post by Giorgio Maone »

Please check latest development build 5.0.8rc4, thanks.
Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:54.0) Gecko/20100101 Firefox/54.0
barbaz
Senior Member
Posts: 10841
Joined: Sat Aug 03, 2013 5:45 pm

Re: [e10s] ABE incorrectly thinks most requests are type "ot

Post by barbaz »

Now INC(OTHER) is matching scripts.

Test page: https://noscript.net/changelog

If scripts are allowed, the changelog will be formatted with bolding and colors. Otherwise the changelog is just plain text.

This ABE rule -

Code: Select all

Site .noscript.net
Deny INC(OTHER)
... blocks the script that formats the changelogs -

Code: Select all

[ABE] < .noscript.net> Deny INCLUSION(OTHER) on {GET https://noscript.net/cl.js <<< https://noscript.net/changelog - 2}
TEST rule:
Site .noscript.net
Deny INCLUSION(OTHER)
*Always* check the changelogs BEFORE updating that important software!
-
User avatar
Giorgio Maone
Site Admin
Posts: 9454
Joined: Wed Mar 18, 2009 11:22 pm
Location: Palermo - Italy
Contact:

Re: [e10s] ABE incorrectly thinks most requests are type "ot

Post by Giorgio Maone »

barbaz wrote:Now INC(OTHER) is matching scripts.
Fixed in 5.0.8rc5, thanks.
At this moment (may change in the future, if/when further load types are supported by Gecko) OTHER matches the following:
  • TYPE_BEACON
  • TYPE_CSP_REPORT
  • TYPE_DTD
  • TYPE_FONT
  • TYPE_INVALID
  • TYPE_MEDIA
  • TYPE_OTHER
  • TYPE_PING
  • TYPE_REFRESH
  • TYPE_WEBSOCKET
  • TYPE_WEB_MANIFEST
  • TYPE_XBL
  • TYPE_XSLT
Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:54.0) Gecko/20100101 Firefox/54.0
barbaz
Senior Member
Posts: 10841
Joined: Sat Aug 03, 2013 5:45 pm

Re: [e10s] ABE incorrectly thinks most requests are type "ot

Post by barbaz »

Confirmed fixed, but it's still not quite backwards-compatible. INC(OTHER) didn't match TYPE_MEDIA or TYPE_FONT before.

Test page for media: http://www.quirksmode.org/html5/tests/video.html
Test page for fonts: https://fonts.google.com/

ABE rule:

Code: Select all

Site ALL
Deny INC(OTHER)
*Always* check the changelogs BEFORE updating that important software!
-
User avatar
Giorgio Maone
Site Admin
Posts: 9454
Joined: Wed Mar 18, 2009 11:22 pm
Location: Palermo - Italy
Contact:

Re: [e10s] ABE incorrectly thinks most requests are type "ot

Post by Giorgio Maone »

barbaz wrote:Confirmed fixed, but it's still not quite backwards-compatible. INC(OTHER) didn't match TYPE_MEDIA or TYPE_FONT before.
They were not documented as "static" ABE types, but fair enough: it was actually in the formerly "rigid" grammar, and if DTD and XBL have their own type, surely discriminating MEDIA and FONT is more useful. Sigh.
Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:54.0) Gecko/20100101 Firefox/54.0
Post Reply